The T1 and T2 cooling packages were drained starting at 14:36, by 14:48 both expansion tanks had dried, meaning the draining rate for both tanks were ~80L/hr, generally it will only take 2 hours to fully drain the system.

Tritium samples were collected after allowing the system to drain for ~5 minutes to clear out water in the drain lines and actually collect samples from the reservoirs. The samples were passed onto RPG for analysis.

The drain valves on the cooling packages and in the BL1A tunnel is left open.

The following maintenance tasks were completed on 2024-11-29 by A. Newsome, A. Kong, A. Tam, M. Dalla Valle:

• Warm cell right window water level topup... the water level was a couple inches lower from the top. Using a tube connected to the nearby sink, the level was topped up fully. See attached before/after picture.
• Hot cell oil level checks:
  • Right side OK. See attached picture.
  • Left side very low compared to July 2022 measurement (suspect internal leak?). The oil was topped up. See attached before/after picture.
• Hot cell scissor table pump oil level check - M. Dalla Valle notes this was topped up approximately July 2022. The table was operated and functioned normally, did not sound concerning. The oil level was visible and acceptable.

Still to do, next maintenance check-up:

• Check HC atmospheric pressure differential gauge
• Lubricate HC turntable (planned for 2025 winter shutdown when target assembly not present)
• Lubricate all telemanipulators

T1-MK1 electrical checks were completed on the hot cell. Specifically the profile and protect monitors were checked (assistance from Micheal Donohoe and Holden Jones from 'probes group'/accelerator systems).

Reference WP C2024-10-30-3.

"WORK INSTRUCTIONS / REFERENCE DOCUMENTS: Perform blip test of target protect module by attaching HV oscillator to protect HV bias. Check that the oscillator signal is readable from U,D, L, R and HALO signal cables.

For profile monitor, connect PM cable assembly to 0518 MWC module in portable NIM crate. Connect 0518 "START" to ch2 of an oscilloscope and "ANALOG" to ch1. Set scope to trigger on Ch2. Connect HV oscillator to profile HV bias. Observe signals on all channel devisions, note any channels not responding."

----------------------------------------

Test outcome (paraphrased from text discussion with Micheal Donohoe)

Profile monitor functioning properly from test, but right 'plate' on protect monitor yielded different results to other plates. May need to be re-checked, pending comparisons with results from previous tests (issues with this because "different oscillator was used"). We may have also not used couplings for cables that were correct for that particular protect monitor plate.

----------------------------------------

In images attached, red cable used for profile monitor (connect to the vertical connector at the rear), the dark green wire is used to ground the system when the profile monitor is being checked. The grey cable bundle (with 'box') is used to check the protect monitor, with vertical pins located by the main lifting flange support bars in the front left of the target (see bottom right of first image).

----------------------------------------

Micheal and Holden returned on the 31st with Thomas Manson, they performed direct testing on the leads of the protect monitor connector (directly hook up leads on protect monitor connector to oscilloscope/multi-meter). All direct checks went well, the waveform issue from yesterday's testing likely resulted from the testing cable having issues.

Electrical testing on T1-MK1 is now complete. Probes group will work to update their documentation and possibly develop a process change to fabricate new cables to make the testing happen fully on the hot cell roof and have it complete faster.

Note: it may be a good idea to perform preventative maintenance on the protect monitor @ T1-MK1 the next time it is on the hot cell. Probes group members noted that the protect monitor components are nearing their end-of-life and it would be good to replace old components to avoid the protect monitor failing in the target station.

A safety walkaround was completed for the Meson Hall Hot/Warm Cells.

The resulting spreadsheet can be found on DocuShare as Document-242733.

Main deficiencies identified:

• Hot Cells:
  • Pressure gauge reading is suspect
  • Operator station phone not working (resolved now)
• General:
  • Lifting equipment has overdue inspection

Action has been taken on all deficiencies.

The active sump high level sensor was tested and correctly alarmed locally. However the alarm in the MHESA RCR lab did not make any sound and no notification was recieved at the main control room. After some investigation, it is believed that the RCR lab alarm 'had already produced a notification at the control room and so no new notification was produced by testing the remote handling sump. Alternatively, the RCR sump alarm may have been silenced in the control room and so the notification was not seen. 

We will look to modify the system so that our sump's alarm does not piggy-back off the RCR sump alarm when notifying the control room.

Secondary flow sensor B1A:T2CS:FGSEC:RDFLOW on the T2 cooling station suddenly became very noisy at around 11 pm, 01 September 2024.

The sensor reading would fluctuate from 0-150 gpm which is beyond typical noise levels for these sensors (~2gpm).

The noise spontaneously ended at around 4 am, 05 September 2024.

The noise likely originated from some stuck debris preventing the paddle wheel from turning normally that got spontaneously dislodged.

Alternatively, changes in the environment temperature could have broken some electrical contact that recovered when the temperature cooled over the weekend. We will make a note to look into this sensor in the coming shutdown and perform preventative replacement of components.

During a replacement of the RCR1 active sump (mechanical services group), approximately 600L of non-active grey water was released into the RH active sump. At this time, this should be all of the water in the RH sump (approximately 600L total).

The RH active sump was drained (over Jul 31-Aug 01), in total approximately 5500 L of water was sent to city sewage.

We also tried to divert water from the BL1A holding tank to the active sump, and learned that we cannot run back-flow through the sump pump. The valving configuration used here was V10, V1, SV1, V6 open - V2, V3 closed. (Note solenoid valve SV1 does not serve a purpose and energizing/powering-down does not affect the observed flow.)

In order to divert water from the holding tank to the sump, we likely need to add a 'T' after V6 to send the flow through the opening port into the sump.

The BL1A holding tank was subsequently drained and is now ready again to receive water from T1/T2/TNF.

A safety walkaround for July 2024 was completed by A. Kong.

Results can be found in the master spreadsheet

Major deficiencies:

• Identified a 1/2 ton chain hoist missing an inspection tag. Mechanical Services will need to be informed about this.

The primary transfer flask was inspected on the 26th of June, 2024:

1. Electrical cabling and connections externally mounted on the flask + on the hook/latch were inspected for wear, damage, proper connections,etc.
   • It was learned that the controls+power cable that drives the latch also runs the height indicator pulley+lead weight system, this should be reworked in the future and replaced with proper load bearing rope/cable.
   • The cable connecting to the main control box was duct-taped heavily, it may be a good idea to replace it with proper reinforcement in the future.
   • The power cable connection to the grey junction box is not terminated properly and should be re-terminated (some other cables may need to be re-terminated properly).
2. Various markings on the exterior of the flask were checked:
   • Marks on the flask flange were still visible
   • Marks on the flask body for reference flask orientation were still visible
   • The weight of the flask (30,000 lbs) had faded and was written over with a sharpie (it may still be a good idea to paint a larger version of the weight so that it is visible by the crane operator)
3. Limit switches throughout the flask was inspected
   • The chain hoist limit switches were tested and both functioned properly, some wear was observed on the spring toggle for the upper limit switch but it does not need to be replaced yet.
   • The limit switches on the hook/latch were tested and functioned properly, with the exception of the 'unlatch' limit switch which we could not test because the unlatch indicator light on the control was broken (this should be re-tested in the future).
4. Rolling/rotating components (pulleys for the door and level indicator, guide wheels inside the flask,  door pin/hinge, main lifting eye, hook latch) were inspected
   • A retaining ring was missing from one of the level indicator pulleys which was promptly replaced
   • All other pins/shafts/rollers were in visually in good condition and rolled properly (despite some being rusty) and had proper retaining components (cotter pins, retaining rings, etc.).
   • Note that for future inspections, it may be a good idea to manually move the various rollers to check by hand if the pins/shafts need to be lubricated
   • Also, the lead weight driving the level indicator should be painted over, and a sacrificial contact pad should be used to prevent the load from wearing the outside of the flask
5. Electrical assemblies inside the control and grey junction boxes were inspected and found to be in good condition
   • *Except for a broken LED for the unlatch limit switch in the main control box (the broken LED issue was sourced to a broken relay that was promptly replaced, the unlatch status light shows up now)
   • All hook/latch actuating switches on the control box seemed to be working (note that the finger-related switches were not tested)
   • Note that there is an 'override' button on the control box whose purpose is unclear, we should investigate this to determine its purpose in the future
   • Note that to access the internals of the control box, only the two central screws along the white line need to be removed
6. The chain hoist and cable pulley system for flask door were inspected
   • The chain lubrication seemed dry, it would be a good idea to re-lubricate the chain properly and perform some form of maintenance on the hoist in the future
   • The chain length is retracted and dropped from a bucket and currently causes the chain to rub against a corner on top of the flask, in the future it may be a good idea to make some changes to avoid this
   • The flask door cables used duct tape to prevent the crimped cable loops from fraying and was still in good condition
   • In the future, we may want to look into the cables' channel to check for fraying or damage along their length.
7. Inspect main flask structure
   • Welds on and around the main lifting eye were in good condition without chipping paint
   • There was visible deformation on the lifting eye from contact with the hook, but this should be no cause for concern, and besides this, no signs of wear/damage was found on the lifting structure
   • The external welds on the main flask body were in good condition
   • The welds on the hook/latch that we could see were in good condition and still fully painted
   • There was some rusting on the platform
   • The welds on the ladder and platform were of lesser quality than the lifting structure but no cracks or broken welds were found
   • In the future, we should determine a weight limit for the platform based on some analysis
8. The aluminum tray was inspected and found to be in good condition, however we should replace the old rubber pieces with new ones at some point
9. The drive mechanism for the doors/hook/latch was tested and we were able to travel the full distance without abnormal sounds from the chain hoist.
10. Seismic clamps were inspected and the bolts + threaded inserts were re-tapped and greased for smoother disassembly/installation.

In addition to the items above, the four 'fingers'/actuators that are holstered by the base of the flask was inspected visually, we will confirm their function before determining whether they need regular inspection/maintenance.

Also, at some point we should inspect the target alignment frames, their camera systems, as well as hot cell flask controls systems.

On May 23, 2024 two sets of plugs from position 1 and 3 and a 12mm Be target from position 5 (serial #109) was removed from T1-MK1 in the Meson Hall remote handling hot cell.

The plugs and spent target are kept in the secondary hot cell (East) on the lift table for future beamspot imaging and to cool down prior to disposal.

In their place 12mm Be targets were placed:

• #107 in position 5
• #110 in position 3
• #111 in position 1

We will continue with leak checking and position measurements in the coming week, at which point this ELOG will be updated.

------------------------------------------------------

UPDATE May 24:

• leak testing began on the T1-MK1, vacuum stagnated at 1.5e-2 Torr and a leak rate of 3.9e-07 atm-cc/sec (system vented, will re-tighten target fittings and re-do leak test). 
• after tightening and repeating pump-down in the afternoon, the vacuum volume stalled again at 1.5e-2 Torr. releasing helium to the target fittings registered a response in the leak tester at all virtually locations (for either side of the ladder). 
• the plan for the coming monday is to re-check the vacuum fittings and re-tighten the swage-lok connections to the targets before repeating the leak test once more.

------------------------------------------------------

UPDATE May 27:

• the swage fittings for the targets on position 1,3,5 were tightened in the morning but the volume still couldn't be pumped down to 1e-03 Torr.
• pumping down only on the vacuum line (up to KF elbow ahead of hansen fitting at outlet of target connections pannel) allowed the pressure to drop to 1e-03 Torr so it was determined that the leak tester + associated vacuum connections were not the problem.
• shortly afterwards, we troubleshooted by pumping only to the 'high pressure test' setting (no Turbo pump) and sprayed helium to various joints above the target flange. we learned that there is likely some leaking through the hansen fittings + the swage elbow from the fittings down through the target flange.
• plastic ferrules are used in this elbow to allow for some seals to be replaced, so this may be the source of the leak.
• then, the fittings on the target ladder were re-tightened and helium was sprayed onto various fittings on the ladder. all but the left join on target ladder #1 (position 9a) does not register a leak rate above baseline after tightening.
• at end of day, the ladder was able to consistently reach a base-line leak rate of 1.7e-07 Torr-l/sec and a vacuum of 4e-3 Torr.
• we will proceed with replacing the water line connections on top of the target flange and re-check the leak-tightness of the ladder in the coming days.

------------------------------------------------------

UPDATE June 14:

several attempts at replacing the plastic (Nylon, Swage NY-1013-1 and NY-1014-1) ferrules and re-establishing the fittings on the target service panel were made since the last update. it was determined that the lines leading from the leak detector to the target panel was leak tight (reached 1e-03 Torr within ~5 minutes of pumping down). a double-male Hansen adapter was made to test this (see image). in the final attempt to re-seal, the nylon ferrul-ed fittings were tightened last and the target managed to pump down to 2e-03 Torr, with a stable baseline leak rate of ~2e-08 Torr-L/s. note that prior to exchanging targets, the leak rate obtained during water flushing was 1e-10 Torr-L/sec. additionally, spraying helium on the inlet fittings still registered a response at the leak detector (~4e-6 Torr-L/sec in the worst case).

we think that the following happened to the target, leading to the leak: at some point the target panel collided with something by the water outlet hansen port (see image), this caused the nylon ferrule at that line to crack/become damaged but no leaks occurred immediately after. when we performed the target exchange, the water line was disturbed in some way either as a result of torque-ing the target fittings or when the ladder was moved on the turntable causing the joint to completely fail and leak. because of the damage from the collision, after taking the fittings apart, it is no longer possible to return the fittings and for them to seal.

the plan moving forward is to implement some minor design changes to 1) replace the damaged fittings on the target panel, 2) better facilitate replacement of the nylon ferrules in the future, 3) better facilitate leak testing on the target service panel connections.

------------------------------------------------------

UPDATE Sep 27:

• After designing, releasing, and procuring the leak fix components (see TRH1681), we implemented changes to the electrical barrel connectors and tested the fit of the cooling line piping (see pictures).
• We checked that the ladder motor, potentiometer, and limit switch circuits all worked properly.
• Some differences were noted:
  • D10414-2 accepts 1/4"-20 screws to mount the panel, not 10-24 like shown in the legacy drawings, the clearance hole on TRH1682 had to be adjusted accordingly.
  • The low profile screws specified for the barrel connectors were difficult to use (item 108 in TRH1681), so instead we used normal 4-40 socket head cap screws and were able to check that the barrel connectors could fully couple without interference from the socket head.
  • We did not use lock washers for the barrel connector flanges because they drop too easily and instead we elected to tighten the nuts well.
  • Some loose leads fell out of the potentiometer (bottom left) and profile monitor (bottom right) barrel connectors. These were not electrically connected to anything so we decided to remove them, they will be kept in a recorded location at the end of the operation. 
• In the coming week, we will look to fasten the water line fully and check that they are leak tight.

 ------------------------------------------------------

UPDATE Oct 02:

• The water line was fastened fully, some items noted:
  • Order of fittings tightened: elbow to bulkhead adapter > elbow to reducer > 5/8" connector to tube > elbow with reducer to tube > nylon ferrule connection
  • We tried to tighten the elbow with reducer to the tubes with the tubes in position to minimize loading the nylon ferrule at the end but found that the outlet tube still ended up protruding significantly from the bulkhead plate (see image). For future operations we should make a note to not load the outlet tube too significantly because it is only supported by the tube (bulkhead nut not fully engaged).
  • The above issue likely resulted from bending the tubes to the 3D printed template but yielding misaligned ends. In the future better templates that emphasize absolute end alignment for tube bending should be made
  • The NPT-Hansen nipple was not tightened before the water line was put on which may have applied unnecessary loads onto the tubes. In the future we will make a note in the drawing on which fittings to tighten first and how to perform the operation to minimize loading on the nylon ferrules
  • In the future some modifications can be made to help relieve this issue for version 2 of the water connections, maybe using flexible metal tubes or thinner walled pipe, alternatively we can implement a more flexible mounting approach for the Hansen ends.
• We noted some tight clearances on the electrical bulkhead. In future implementations of the fix, the connectors can be spaced farther apart or staggered, also they should be moved further down from the top flange of the target ladder.
• Leak testing proceeded with the following results:
  • Total time under vacuum ~2 hours
  • Reached 'fine' vacuum within 1 minute, pressure saturated to 2e-3 Torr within 3 mins 30 secs.
  • Baseline leak rate of 0.0e-10 Torr-L/s reached within 7 minutes of pumping down.
  • ~1.4e-9 Torr-L/s peak @ 15s delay (to first signal) after 1s x 3psi He to general vicinity of newly made water line connections << deemed acceptable
  • 0.5s x 3 psi bursts of He
    • ~0.9e-09 Torr-L/s peak @ 15s delay to nylon ferrule outlet line (did not change after further tightening) < deemed acceptable
    • ~0.2e-10 Torr-L/s peak @ 15s delay to nylon ferrule inlet line < deemed acceptable
    • ~0.5e-10 Torr-L/s peak @ 40s delay to NPT-Hansen outlet < deemed acceptable
    • ~0.4e-10 Torr-L/s peak @ 40s delay to NPT-Hansen inlet < deemed acceptable
  • SS swagelok connections not tested thoroughly due to low likelihood of being a leak source (all tightened properly to 1.25 turns as per swagelok directions)
  • He 'bleed' test by closing hot-cell He wand solenoid valve and bringing wand opening ~5 cm from pos.5 target swagelok resulted in a stable leak rate of 0.8e-8 Torr-L/s << target ladder leak likely prevents pressure in target ladder from bottoming out.
• Will carry on with leak testing the target ladder and possibly the swagelok fittings on top of the hot cell if necessary by the end of the week.

 ------------------------------------------------------

UPDATE Oct 04:

• The target ladder was pumped down at ~11 am and reached the same baseline as yesterday (0e-10 Torr-L/s @ 2e-3 Torr within 3mins, 30 sec), but in the afternoon, the baseline leak rate worsened to 1.0e-8 Torr-L/s.
• For reference, 0.5s of 3psi He was delivered to the water outlet nylon ferrule joint and registered a response of 1.8e-8 Torr-L/s with a delay of 20 sec. A 3s spray at 3 psi around the water fittings registered a 4.9e-8 Torr-L/s response with a 15s delay.
• The following was observed from sending 0.5s of He to the target ladder Swagelok fittings (see Img for more details):
  • Leak rate baseline improved during testing (~over 1 hour) to 3.0e-9 Torr-L/s
  • Bellows, and targets 2-5 registered leak rates in the order of 1e-8 Torr-L/s
  • Target 1's left Swagelok fitting registered the highest leak rate at 2.8e-7 Torr-L/s (when the leak rate baseline would have dropped to the lowest value of 3.0e-9 Torr-L/s)
• We checked the KF and hansen fittings to make sure that they are not a leak source and managed to bottom out on both the leak rate and pressure readings on the leak tester.
• We will take off Target 1 in the next session, run a die over the male threads, clean the conical face of the male thread with scotchbrite,and attempt to re-do the target joint before leak testing once more.

 ------------------------------------------------------

UPDATE Oct 11:

• We removed target at pos 1 (ID 111), noticing that the swagelok nuts were easy to undo.
• Working theories on how they became 'loose' include: 1) creep undoing misalignment between the two swagelok tubes on target, 2) vibration from the nuclear ventilation onto the target ladder undoing the joint, 3) thermal changes between summer and 'fall'.
• The threads on pos1 seem worn, it may be a good idea to plug this ladder position permanently to avoid further damaging the threads and jeopardizing the ladder in the future (see image).
• The ferrules on the target seem to be in good condition but we should still clean them with the scotch brite tool
• We added plugs over the exposed male swagelok ports to protect from debris entering the water line.
• We placed the removed target between two wypalls and weighed the wypall down with a wrench to protect it from debris.
• The 3D printed scotch brite tools need to be re-designed for 3/8" drive size (was designed for 1/2" drives).
• We will return to bag, and package the target for safe keeping in the secondary hot cell; clean the male swagelok threads and sealing faces on pos 1 target ladder; and possibly place permanent plugs in pos 1.

 ------------------------------------------------------

UPDATE Oct 15:

• The leak tester was calibrated with an external calibrator. Original leak rate 1.3e-7 Torr-L/s, 2% loss over ~10 years >> ~1.1 Torr-L/s leak rate.
• The leak checker registered an initial leak rate of ~2.3 Torr-L/s before calibration so we corrected the measurement (at least within the calibration range) by a factor of 2.
• This voids old leak testing data that we made on the target ladder.
• We will add a note to the procedure to ensure that the leak detector has been calibrated before performing leak testing in the future

 ------------------------------------------------------

UPDATE Oct 23:

• After some testing with 3/8" tube Swagelok threads outside of the hot cell, we determined that the reason the threads in pos 1 of the target ladder was 'difficult' to run was likely because the nuts were over-tightened previously.
• There is a risk of compromising the threads if we run the die over over-tightened threads because they have been shifted from where they should be. We will instead attempt to replace the plugs, after cleaning the sealing face on the male threads using the cleaning tools.
• We also checked with a Swagelok tightening gage (3/8) to see whether the other targets were tightened properly, and all had gaps smaller than the gage. Note that we could not get fully into the gap with the gage on the side by the profile/protect monitor due to interference with a support column with the gripper. We will make an extender tool to avoid this issue in the future.
• We also bagged the target that was previously mounted onto pos 1 on the ladder and moved it to the secondary hot cell for storage (temporary).

 ------------------------------------------------------

UPDATE Oct 24:

• The threads and sealing face on pos 1 were cleaned with scotch brite, blown with compressed air from the inside by connecting compressed air line at the service panel, blown with compressed air from a bottle externally, and wiped with a wypall with the manipulators.
• The old plugs were moved to the secondary hot cell, new plugs were tightened onto pos 1.
• Initially the baseline leak rate was 3.3e-8 Torr-l/s at 4e-3 Torr. With He, we were able to determine that the plugs, specifically position 9a, was the most leaky joint in the ladder.
• After a couple rounds of tightening the plugs, and making sure that the remaining targets were also tight, the baseline improved to 1.7e-9 Torr-l/sec. The plug at position 9a also no longer registered a leak response from having the He line brought up to the joint.
• The plug at position 9a still is the leakiest location in the ladder however, registering a peak leak rate of 2e-8 Torr-l/s at the final baseline.
• We deem that the ladder is sufficiently leak tight, and we will move on to replacing a rubber coupling on top of the target ladder before performing target position measurements.

 ------------------------------------------------------

UPDATE Nov 25:

• The coupler was successfully removed with a custom puller tool (see TRH1730, TRH1731, TRH1732, and Document-244878). This tool is used for the 3/8" shaft side.
• The press fit coupler had to be cut up with a dremel and saw before using the tool.
• The old coupler had to be 'cut-up' to be removed. During initial troubleshooting, the rubber body split, indicating that it was near end of life.
• The replacement coupler system could be inserted without removing any parts.
• The relative alignment of the shafts may have changed up to 1/2 a turn during this process (the printed clamps were not strong enough to resist torques exerted when attempting to remove the press fit coupler).
• Potentiometer table will need to be reviewed when performing target measurements.
• Light loctite to be added to the treads.
• Note: the old coupler has stainless shaft adapter (3/8" step down).

 ------------------------------------------------------

UPDATE Nov 28:

• Loctite added to all coupler screws (including flexible coupling from motor to three-way gearbox)
• Final leak check completed on replacement piping on service panel with the following results
  • baseline leak rate bottomed to 0.0e-10 Torr-L/sec within 7 mins and pressure bottomed to 1e-3 Torr within 5 mins.
  • baseline leak rate worsened to around 1e-9 Torr-L/sec by the end of leak testing, assumed due to increase in amount of trapped He in the line through testing
  • (time to detect [s] || peak leak rate [Torr-L/sec]) values for 3psi, 0.5s Helium using wand on top of hot cell
    • inlet side:
      • 20s || 3.8e-10 for nylon ferrule joint
      • 20s || 6.3e-10 for o-ring flange
      • 20s || 2.7e-10 for hansen NPT thread
    • outlet side:
      • 15s || 3.5e-10 for plastic ferrule joint
      • 15s || 5.1e-10 for o-ring flange
      • 20s || 0.8e-9 for hansen NPT thread

The target ladder was moved to the lift table in preparation for measurements.

 ------------------------------------------------------

UPDATE Jan 03, 2025:

• Measurements (elevations) of the targets and alignment jig were completed over December to early January.
• Notes on measurement procedure:
  • Used the old alignment jig instead of new (easier to insert/remove)
  • Noticed that the target ladder potentiometer is jumpy at around 12 kOhm
  • Placed the jig between positions 4-5 throughout (pot reading of 11.65 kOhm places the jig in beam height - in line with protect monitor)
  • For left-right measurements, moved the alignment jig 'into-position' (i.e. set potentiometer value to 11.65 kOhm) to 'zero' the micrometer before changing ladder position (potentiometer value) back to the target being measured (turntable height not touched after zero-ing off to the jig)
    • For the entry side, because the protect monitor is obstructing, target 5 had to be measured above the protect monitor (jig also zeroed off above the protect monitor, i.e. not at the resistance value given above), while all other targets were measured below.
    • This introduces some error since we are measuring the target not in the beam position.
  • Target height measurements were done with measuring tape as normal.
  • Targets were measured with turntable flange having an angle of ~0.2 degrees, plumb bob and jig measurements taken with turntable flange 'leveled' as best as possible (angle of ~0.02 degrees)
    • For future measurements, the turntable should be also leveled before target measurements.
  • Levelling the turntable flange was done with a Digi-pass digital level (DW-1300XY, in absolute level mode, calibrated before starting)
  • By the cutout, the turntable flange is less stiff, causing level measurements to be tilted inward. We therefore relied more on level measurements on solid portions of the turntable flange as well as on the target flange.
  • Levelling bolts could be turned with target ladder seated.
  • If the turntable is not rotated, and only raised/lowered, the levelness of the table and target remained approximately constant (e.g. from 0.02 degrees change to 0.04 degrees as the turntable is lowered fully from fully raised).
  • If rotated, the turntable level changes significantly (e.g. from 0.02 degrees to 0.15 degrees)
  • The flange was re-levelled between entry side and exit side measurements for the plumb bob and alignment jig measurements
  • With new nuclear ventilation filters, flow inside the hot cell perturbs the plumb bobs and prevents them from settling.
  • Nuclear ventilation had to be turned off momentarily for the plumb bob measurements
  • Nuclear ventilation is returned when working on the hot cell roof
  • Nuclear ventilation is controlled by turning the toggle on the electrical disconnect panel from 'hand' to 'off' ('auto' setting assumed not used)
  • Respirator and lab coat worn throughout plumb bob measurement job, air survey done by RPG after job (confirmed OK)
  • Area also 'taped off' during plumb bob measurements to prevent non-worker entry
• The coupler exchange resulted in around a 0.4mm shift in the target heights
• A shoulder screw and wood clamp is used to lock the plumb bob jig in place (old screw/nut missing, no other components in toolbox provides locating fit for pivot joint on jig)
• Assumed new plumb bob measurements more accurate due to better/more accurate 'levelling' equipment (digital level).

At ~8am on May 06, the expansion tank level sensor for the T2 cooling system suddenly became noisy. 

Cyclotron fault ref: 16915

Approximately 11:00 am today, the noisy sensor was replaced with a spare, upon which it was learned that the spare sensor is broken (registers 0 level and not detected on PC through USB adapter).

The old (noisy sensor) was then replaced at around 12.00 pm onto the expansion tank and it was found that the noise had subsided.

A possible explanation to the noise would be loose connections/grounding wire.

We will continue to monitor the sensor in the coming days, and order replacement sensors has been placed and we will be able to replace the sensor soon should it become noisy again.

Edit 2024-05-07 - A. Newsome: EPICS monitoring shows the sensor appears to be behaving normally since the aforementioned events. Most likely attributed to improper grounding. The fault will be closed. See attached screenshot.

As of 2024-04-03, the BL1A Holding Tank water level is approximately 580L. It is recommended to drain the tank prior to the start of the operating year.

The lower amplifier for the T2 target station air supply was removed for teardown, inspection, and rebuild. 

The goal of this operation is to understand wear development in the air amplifier over ~13 years of operation, and potentially determine a recommeded service interval.

---------------------------------------------------------------

The T2 volume was vented for an unrelated maintenance operation during this time.

During testing, prior to removing the lower amplifier, both regulators were set to ~20 psi. 

The upper air amplifier resulted in ~125 psi at the outlet while the lower air amplifier resulted in ~120 psi at the outlet.

The upper air amplifier had more audible air flowing out from the muffler than the lower amplifier. 

'Scratching' sounds in both amplifiers were comparable.

The following cycle times were recorded with the M20 BB raised/out (min:sec):

    UPPER: 1:22 / 1:00 / 2:06

    LOWER: 2:30 / 1:46 / 2:12

The following times were recorded to raise the M20 BB (sec): 

    UPPER: 8.36 / 8.76

    LOWER: 8.56 / 9.10

These times will be compared against after completing the teardown and rebuild of the lower amplifier, at which point this ELOG will be updated.

-----------------------------------------------------------------

UPDATE (Feb 23, 2024):

The lower air amplifier was serviced (photos in 'S:\Albert Kong\Shutdown Files\2024\Feb20 T2 Lower Amplifier Cleanup'):

1. Full assembly cleaning.
2. Piston and barrel was cleaned and lubricated, o-rings and plastic components replaced (with new lubricated ones).
3. Pilot valve components replaced (except plug).
4. Check valves (4x) cleaned and plastic/rubber components replaced (with new lubricated ones). 
5. Muffler cleaned (blown out with compressed air).
6. Spool and sleeve assembly cleaned, o-rings replaced (with new lubricated ones), and rubber stopper replaced.
7. Clamping rods tightened to ~17 ft-lbs. 

Note: the piston o-rings were difficult to seat on the piston body/teflon ring. During assembly, we instead seated the o-ring in the barrel on the piston plates (see picture), which allowed the oring to be seated properly, before placing the piston body onto the piston rod.

After servicing, the amplifier was returned to the station, air connections reconnected, and tested. 

Note: it is recommended to do torque-ing of the clamping rods as a final step to simplify mounting of the amplilfier and re-doing connections to the rest of the compressed air system.

The first observation we made was how silently the lower amplifier now operates when cycled: only the exhaust sound from the muffler can be heard.

Note that the piston's motion can be heard when listening ~5cm away from the amplifier barrel. 

The following times were recorded to raise the M20 BB (sec):

    UPPER: ~8.5 

    LOWER: ~7.7s

The following cycle times were recorded with the M20 BB in the out/raised position (min:sec):

    UPPER: 1:05 / 1:04

    LOWER: 1.22 / 1:45

The outlet pressure from the amplifier registers 120 psi with the regulator set only to 15 psi (improvement from previous performance as well as the upper air amplifier's performance).

We will check in on the amplifier next week to see if it still operates silently and can actuate the beam blockers/profile monitor without issue, at which point this ELOG will be updated. 

-----------------------------------------------------------------

UPDATE (Feb 26, 2024):

The lower amplifier was inspected this morning, higher volume sound could be heard from the drum in concert with the motion of the piston, but still much quieter than before servicing.

Feb 1 2024

Hand Pump observed to be losing pressure at full extension of cylinder, seals likely blown. Due to Enerpac P84 being an obsolete model, a newer model hand pump was ordered.

Feb 15 2024

Hand pump (Enerpac P84 Ultima) installed at Indium Extruder Station, observed to work properly in both directions of flow, deemed operational. 

 The recently removed resin can from the T2 target station was rinsed with city water and flushed with compressed air for 1 cycle. 

We will run it through a few more cycles before proceeding with drying under the fume extractor, at which point this ELOG will be updated. 

UPDATE (Feb 08, 2024): The T2 resin can underwent 3 more air cycles. Today it was moved under the fume extractor with the lid cracked open and will be left to dry till next shutdown. It is worth noting that some clumps of old resin was found in the secondary resin can which was previously under the fume extractor. This resin should be disposed of alongside the T2 can's resin once dried.

The recently removed T1-MK1 target ladder was flushed with city water for ~30s at all target positions. It was then purged with compressed air at ~5 psi for ~1 minute at all target positions. We then left the target ladder in pos5 to fully extend the bellows and allow it to dry overnight.

This ELOG will be updated as we progress with work done to dry the target ladder and eventually exchange the spent targets.

-----------------------------------

UPDATE (Feb 02, 2024): in the morning, the target ladder was moved to pos4, and air was allowed to run through the morning, in the afternoon the ladder was moved to pos2 and air was ran. At the end of the day, the ladder was moved to pos 0 and air was ran through over the weekend.

-----------------------------------

UPDATE (Feb 05, 2024): the following procedures were completed.

1. the air connections on the target ladder was replaced with the leak test/vacuum connections as per instructions.
2. LN2 was taken from the ISAC-I facility and used to perform two pump-down cycles with the Agilent He leak detector + cold trap.
3. First pumpdown cycle: 3min45sec to go from roughing to fine pumping, left for a total pumpdown time of1hr20mins, a relatively small amount of ice formed on the cold trap cylinder (see image).
4. Second pumpdown cycle: 2min30sec to go from roughing to fine pumping, 4min30sec to reach 1e-3 Torr, 5min to reach < 1e-3 Torr. 
5. Due to good pumpdown in the second cycle, we proceeded with leak testing: Baseling 1.0e-10 Torr-L/sec | ~1 sec He applications at 3.5 Psi directed at fittings/welds/bellows. Worse case measured leak was 0.9e-7 Torr-l/sec in pos 13 of fig 12 of Document 46600 (see taken image for detailed measurements at various locations).

We will prepare an update to the procedure to better log the leak rate measurements and update this ELOG/the procedure. Note that ice removed from second pumpdown was actually comparable to the first (see image, ice knocked off during cold trap removal), but total pumpdown duration (including leak testing) was ~4 hours for the second pumpdown. 

 UPDATE (Feb 13, 2024): see attached record with new template.

 The following tasks were completed:

1. Yesterday (Jan 18), the T1/T2 cooling packages were inspected for leaks and some moisture was found underneath the CUNO filters behind the resin cans on both T1/T2. It was determined that the source of the leak was a loose fitting on the drain valves of the CUNO filter housing (see photo). Today we confirmed that no more moisture is found underneath the CUNO filter after these fittings were tightened. A note will be made to keep this in mind for future shutdown operations.
2. The shaft couplings on the T1/T2 target ladder drive system were inspected: no cracks were identified on the rubber couplings.
3. Actuation of the T1/T2 profile monitors were tested. Vacuum levels at T1/T2 remained stable throughout (see picture).
4. The pneumatic connections/hose/fittings for the T1/T2 profile monitors, and M9/M20 BB were inspected - therere were some air leaks out from the ASCO valve + pneumatic flow control valve on the T2 profile monitor when it was in the down/out position. The same leak out of the pneumatic valve by the profile monitor connection was found at T1. Given that both profile monitors actuated properly however, these leaks were determined to not be an issue.

Next week, we will continue work by actuating the M9/M20 BB at T2, at which point this ELOG will be updated.

UPDATE Jan 30, 2024: The M9 and M20 beam blockers were tested, no leaks through the tubing were observed, actuation was quick, ops confirmed no perceptible change in 1A vacuum levels during actuation and EPICS reads correct blocker position signals.

Note: the M20 blocker was actuated by ops, while the M9/T2 blocker was actuated using the ASU on the B2 level of the Meson Hall. The ASU did not require the area to be blocked off to actuate the T2 blocker.

------------------------------------

Note: Ops noted that the profile monitor interlock status has to be reset if a timeout is encountered (such as if the pump were to be turned off or a trip was encountered) before the profile monitor can be actuated (see picture).

Note: When inspecting the system, we tried to open the rotary collimator air supply and the handle on the pannel mounted valve broke off (like it did for the T2 valve last shutdown - see picture). 

UPDATE Jan 30, 2024: The handle to the rotary collimator air supply valve was replaced (see image).

Note: There was also some small leaks at the hose clamps at the T2 profile monitor, in the out position. This leak was however smaller than the leak through the valve. 

 The following tasks were completed in preparation of re-filling the T1 and T2 cooling packages:

1. All valves were returned to the 'open' configuration (note, compressed air to collimators at T1 is to remain closed - rotating collimator no longer in use || also note, odd handle configuration at T1 panel when all valves are open - see attached).
2. Leaks from valves were checked, two leakly valves found at T2 (Swagelok SS-44S6 and SS-45S12), the stem of these valves were re-tightened and leaks were remedied (see picures).
3. Spent CUNO filters in buckets were dealt with: water drained to MH HC active sink, and filters organized into a single 'dry' bucket.

The cooling packages will be filled to 38 cm at the expansion tank tomorrow, and the pumps turned on to check for leaks at pressure, at which point this ELOG will be updated.

UPDATE:

The cooling packages were filled on Jan 16 to ~38 cm, after which both pumps were turned on and a few more leaky valves were identified at the T2 station (none at T1).

We remedied the leaks by tightening their stems and will continue to monitor both systems for a few days.

Besides the leak, we found that the TGT:OUT pressure gage was incorrectly reading 0 psi. We will be entering the 1A tunnels to drain the 1A holding tank soon, at which point we will look to remedy this issue (either a valve was left closed, or an air column is trapped at the gage port).

UPDATE (30 Jan, 2024): TGT:OUT pressure transducer at T1 was successfully replaced and is now reading the correct pressure (see image, ~20 psi is about the same as readings from 2021 when the sensor was working properly).

The resin flask on the T2 cooling package was exchanged today (new resin can prepped in the morning). 

The newly installed resin flask was placed slightly off-set from the marking on the platform to ensure that the braided hose will clear the blocks when replaced at the end of shutdown.

The spent resin can was dropped off in the hot cell, ready to be prepped for drying. 

See attached image for illustration of the rigging solution used, ~20 ft was covered by two sligs and a shackle to clear the MH mezzanine.

 The following tasks were performed:

• All electrical, water, and pneumatic connections to the T1 target was reconnected.
• 3x and 2x CUNO filters for T1 and T2 respectively were changed (O-ring for the housings were not changed).
• All hansen fittings on the front mounting plates of the T1 and T2 cooling packages were inspected.
• All proteus paddle wheels (Q1-5) for both T1 and T2 were replaced. 
• The main reservoir drain valves on both cooling packages were closed in preparation for re-filling.
• All other ball valves in the cooling packages were returned to open, also in preparation for re-filling.

Two buckets containing (primarily) active water from the CUNO filters were brought closer to the boot up area, to be moved to the active sink in the MH hot cell lab in the coming days (pending ELOG update). 

Spent shafts, paddle wheels, and o-rings from the proteus flow meters will be checked and appropriately tagged/disposed of, also in the coming days.

------------------------------------------------

Some observations for future shutdown work:

• The drain port at the bottom of the CUNO filter housings can be used to verify that the filter line has been successfully closed off.
• Prepare several lengths of flathead screws to service the proteus flowmeters.
• Prepare spare locknuts and screws for the proteus flowmeters in case they become stripped or a locknut is dropped.
• Stick to well-fitting gloves to make using screwdrivers, picking screws, etc. easier.
• Possibly a pipe or coupling could be used with the tool for the CUNO filters to make it easier to loosen the CUNO filter nut in hard to reach areas.
• If possible, it may be a good idea to replace the o-rings on the CUNO filter housings in future shutdown operations (dash # to be identified). 

T1-MK1 was moved from the target station to the hot cell roof. 

Radiation surveys report 140 mSv/hr at 0.5m for the bare T1-MK1 target ladder.

On the hot cell roof, fields are 220uSv/h at 0.5m.

T1-MK2 was subsequently transported from the storage pit to the target station. 

After the transport operation, the 1A vacuum volume was then pumped down, and reached approximatley 1/10 of the vacuum level prior to venting.

Update: a few hours after the vacuum volume started pumping down, vacuum levels stabilized to the level observed prior to venting (see image) - confirming that the o-ring seal on the target is good and that the transfer operation was a success. 

------------------------------------------ 

During the transport operation we noticed that the chain drive on the target transfer flask was slipping considerably more when powered through the power bar on the camera stand. 

This issue was remedied by powering the transfer flask through a dedicated plug, and will be noted in procedures for future transport operations.

Jan 10, 2024:

The cooling packages for T1/T2 was drained this morning. ~150 L from each of the cooling package reservoirs + expansion tank was drained to the BL1A holding tank. The holding tank held ~400 L initially and is therefore now filled to ~700 L (out of a 800 L capacity). We will likely drain the holding tank in the following morning at which point this ELOG will be updated.

---------------------------------------------------------------

UPDATE (Feb 01, 2024): After recieving instruction from RPG to drain the holding tank to the active sump (due to delays in completing Tritium sample tests), we mistakenly drained ~50L of the 1A holding tank water to the city sewage system (initially it held ~610L and after the draining it read ~580L). This error was caused by misinterpretation of the 1A tank draining systems/procedure. In fact, it is uncommon for RH to pump active water from the 1A holding tank into the sump. It is also unclrear whether a valve configuration to do so exists (if it does, no flow reading can be taken). 

However, because of this error, we learned that the plumbing to the city water drain is slightly blocked (see image).

Our plan of action moving forward will be to wait for the Tritium results and coordinate with RPG to determine the appropriate next steps.

 ---------------------------------------------------------------

UPDATE (May 02, 2024): The active sump level was inspected visually on April 29 and it was noticed that it was close to full. 1x 1L and 2x 20mL samples were then collected and passed to RPG for testing. When RPG completes their Tritium analysis, we will drain both the holding tank and the active sump. See this link.

 ---------------------------------------------------------------

UPDATE (Aug 06, 2024): The tritium analysis was finally completed for T1/T2, since the sump only received high active water from the removed T1 target ladder, tritium analysis for the sump water would have been covered by the T1/T2 analysis. Gamma spectroscopy was done for the active sump (here). The draining procedure will be recorded in a separate ELOG.

 See Cyclotron fault 16553. 

"B1A:T2CS:FGTGT readback is toggling at the warn limit of 3.0 GPM and over the last week has started crossing over the trip threshold of 2.8 GPM, tripping off the water package. Initial Action Taken: 1A is scheduled to take beam on December 13."

Upon reviewing the flow trend over the past semester, the flowrate has been hovering around 3.0 GPM the whole time.

Meanwhile, the water temperatures in/out of the T2 target has remained stable between 24C to 31C with the warn and trip limits > 35C. 

It should therefore be safe to run the target cooling water at a lower flowrate.

As a temporary solution, the low trip limit was adjusted to 2.5 GPM. While the warn was kept at 3.0 GPM.

We will look at how flowrate through the target can be increased in the coming shutdown.

For reference, the T1 target flow trip was set to 2.0 GPM an warn was set to 2.3 GPM (see MH-RH ELOG 331).

UPDATE(June 18, 2024):after shutdown service and replacement of the proteus paddle wheels, flow through the target seems to have recovered and is stable at around 3.7-4 gpm.

Today we re-lubricated the o-rings on the top air amplifier for T1/T2, and replaced the o-rings on the bottom air amplifier with new and also lubricated o-rings. 

We used Haskel lubricant #50866.

The bottom air amplifier spool assembly was noticably dirtier than the top (see attached). and we noticed some worn spots on the o-rings. This was not the case with the top air amplifier spool (it has not been used outside of testing since being fully serviced in 2012).

Replacing the o-rings did not result in any perceptible change to the cycle rate and sound of either air amplifier (~50 sec/cycle, same sounds made by the piston before and after servicing). 

We initially planned to also remove the sleeve on the amplifiers (see image) but could not do so easily. We will contact Haskel to get some input regarding how best to remove the sleeve.

The plan is to perform a full teardown of the lower air amplifier next shutdown (2024).

The level sensor for the active sump in the RH lab was tested.
The level sensor is working properly and is operational.
 

The water level in the T2 cooling system expansion tank dropped to below the trip treshold of 20 cm (ref Cyclotron fault #16123), requiring the beam to be 'defined' off (see attached image).

Some calculations (see attached .html) show that over the period in which the tank level was dropping (from July 12 - Aug03, 2023) approximatley 11L of water was lost. At the end of the 'drop period' the leak rate was at a maximum of approximately 1L/day or 40mL/hour. 

The tank was filled up to 39 cm and we will continue to monitor the water level in the coming days.

We will also plan to enter the BL1A tunnels in the coming maintenance day (Tuesday, Aug 08) to check for poolig water. 

Alternative to a leak, entrapped air in the system may have escaped/been displaced to allowed 11L of water from the expansion tank to drop into the cooling loop.

If the lost 11L indeed occured due to leak, at least the leak will be outside of the vacuum volume since vacuum levels remained stable.

A likely source for a leak would be one of the exchanged valves from last shutdown.

Alternatively one of the not-exchanged valves may have started leaking due to aging/radiation damage.

--------------------------------------------------------------------------------------------------------------------------------------------

 UPDATE: Aug 08, 2023

The expansion tank at T2 was topped up to ~44cm on Friday Aug 04 ~12pm.

Over the long weekend (Friday Aug 04 ~12pm to Tuesday Aug 08 ~7am), ~14cm of water was lost and the water level in the expansion tank went below the low level warning limit (30cm).

Looking at the water level trend, the leak rate seems to be increasing (see attached - rate approximately doubled/trippled to 3L/day or 120 mL/hour).

Maico and Albert entered the BL1A tunnels at Aug 08 ~1pm and found trickling water underneath the T2 cooling package (see attached). Fortunately, no pooling water was found in the BL1A tunnels.

A plan must be developed to decide the appropriate course of action to remedy this problem.

Tentatively we have two choices:

1) attempt to remedy the leak during a maintenance day by uncovering the T2 cooling station. 

2) accept the leak until the mini shutdown in October (requiring ~2-3 expansion tank 'top-ups' every week for 2 months or so).

--------------------------------------------------------------------------------------------------------------------------------------------

UPDATE: Aug 11, 2023

The expansion tank was topped up again to ~44cm on Thursday Aug 10, ~8am.

Since then the water level has dropped to ~34.5cm on Friday Aug 11, ~9am. ~10cm of water was lost within the span of a day, equating to ~6L/day or ~250mL/hour (see attached calculations).

From the data, it seems like the leak has stabilized to this value. 

An SAS job request has been filed to uncover the cooling package during the mini shutdown to fix the leak. Leading up to this, it would likely be a good idea to regularly enter the BL1A tunnel and assess the condition of the leak. 

Additionally, it will be necessary to fill the expansion tank to the brim every two days or so to keep it from tripping the beam.

--------------------------------------------------------------------------------------------------------------------------------------------

UPDATE: Aug 15, 2023

We took advantage of an unexpected maintenance window this week to uncover the blocks surrounding T2 and identified that the source of the leak was a 'pinched' o-ring on the Q2 flow meter (demin water return - see attached pictures). 

The bottom set of screws on the flow-meter o-ring plate was loose when we took it apart. The o-ring may have been pinched when it was assembled back in 2022, making it difficult to establish even loading on all screws.

The bottom screws then creeped loose over time, creating the leak. 

The o-ring was successfully replaced, the pump was turned back on with no immediate leaking at the service flow-meter, and the expansion tank was filled to 39cm.

We will monitor the water level overnight and inspect the cooling package for leaks before deciding the next steps tomorrow morning (if no leaks found, we will proceed with closing up the T2 area).

We will specifically asess whether the puddle underneath the main tank (see picture) dries up in addition to tracking the expansion tank water level.

--------------------------------------------------------------------------------------------------------------------------------------------

UPDATE: Aug 16, 2023

The water level remained stable overnight, up-close visual inspection of the serviced flow-meter and the cooling package in general confirms that the leak has been fixed.

The fill rate of the active sump in XTpage P2 also leveled.

See attached html document (updated calculations and notes) for relevant information.

Work to re-place the blocks started after confirmation of the fix.

Between 1:00 pm and 6:00 pm on July 17, 2023, the T1 target cooling water flow rate sensor exhibited some noise which resulted in it dropping below the warning/trip limit. The control room requested that the threshold values be lowered - permission was granted. The thresholds were set to 2.3 gpm for both warn and trip levels. See attached screenshot from EPICS sensor data. The occasional spikes and dips continued throughout the night fairly consistently.

While the cooling water temperature (TCOUT) fluctuates, there does not seem to be an obvious correlation between the temperature and the flow rate, which suggests that the flow rate is still adequate.

It is worth noting that overall, the FGTGT signal has been slightly trending downward since around mid-July (see attached pic).

It is suggested to replace this sensor as soon as possible - perhaps the mini shutdown.

On May 31, the low active water in BL1A was turned off briefly and upon re-starting, the flow to the T1 heat exchanger was very close to the reported warn limit (~6 gpm). 

To see if this low flow could be remedied, we entered the BL1A tunnels briefly on Jun 06 but found no flow regulator valves for the T1 heat exchanger. 

The low flow to the heat exchanger after the secondary water pump was re-started may be a normal characteristic of the system. Alternatively, this may have been caused by entrapped air in the heat exhanger after it was drained during 2023 shutdown to replace a leaky valve. 

The latter is grounded in the fact that the flow in T2 recovered immediately after the flow was re-started. T2 was drained through a drain line in the tunnels whereas T1 was drained at the cooling station through a filter/copper line. 

The difference in draining methods may have resulted in entrapped air and thus flow issues in T1 but not T2. 

See Cyclotron Fault 15787. The target ladder motor failed to move to the commanded position during a test. The T1 area was uncovered and the motor assembly was inspected by M. Dalla Valle, A. Newsome, and A. Kong. It was determined that the rubber coupler which connects the motor shaft to the drive assembly had degraded and snapped. This is similar to an incident which recently occurred on T2. The coupler was replaced with an all-metal version. The system was tested by the DCR operators. They ran the target ladder to each extreme (position 5 to position 0) multiple times. The target ladder was also moved to a few positions which had targets installed, and the cooling package was energized to run water through and confirm functionality. All feedback sensors for the target ladder assembly and cooling package were observed to behave normally, as confirmed by the DCR. RH group confirmed visually that there were no observable issues. The system is now considered operational.

It is recommended to change all drive system coupler parts on both T1 and T2 during the next shutdown to prevent this issue from happening in the future. These parts should be inspected during the annual inspection of the T1/T2 assemblies as well.

The HEPA filter in the nuclear ventilation system for the Meson Hall Hot Cell was changed and tested by mechanical services under work permit C2023-04-25-14.

They have given approval to operate again. The system is re-energized.

The T1 motor controller hearbeat tripped (noticed this morning). 

This does not seem prevent beam delivery or target ladder actuation (see screenshot, target still ready for beam).

An improperly functioning hearbeat led on a controller may indicate that the controller is beginning to fail.

Interestingly, the profile monitor 'out' status registers a warning - the two may be related: i.e. some electrical work interrupted the two signals. 

This problem will be looked at in the coming days.

(Update) 

After consulting with Tony Tateyama from Cyclotorn Controls group, the two trips were re-set and the controller heartbeat is now 'green' again.

The motor controller is likely still healthy, seeing how it was installed only a few years back (~2015). The cause of the trip should be some electrical work on the mezzanine.

(Update - May 08, 2023)

The controller HB tripped again (noticed in the morning). Coordinated with operators to have it re-set. This was done automatically by driving the target ladder to position 2 then back to position 0.

(Update - May 16, 2023)

The controller HB on T2 tripped, and re-set by ops. 

We tested the air amplifiers by actuating the beam blockers and profile monitors on the T1 and T2 target stations. 

The blockers and profile monitors were each actuated (brought up/out and down/in fully) > 5 times and vacuum remained stable throughout (see attached).

The cycle rate of the air amplifiers (time between 'puffs') under steady conditions (target devices kept in the out position) was measured to be between 60-90 seconds, with 90 seconds corresponding to when the 'flow control' ball valve is fully shut and ~60 seconds when the valve is opened (either partly or fully). 

When a target device is brought up, a cycle event will occur shortly after the device reaches the out position.

This cycle time is much longer than last reported in 2014 (see elog #119), by a factor of 2-3. 

The top amplifier sounds different from the bottom amplifier (cycles between being 'rattly' and 'quite' between cycles).

Seeing how the beam blockers and profile monitors were actuated smoothly with the air amplifiers, we will keep the system running off the lower air amplifier with the 'flow control' ball valve kept open (not shown).

At some point we will look into disassembling the top amplifier to see why it sounds differently between cycle events.

For reference, both amplifiers read 20psi at the regulator and 120 psi to the target station. 

 We tested the actuation of the M9/M20 beam blockers on the T2 monolith (3-4pm).

Vacuum levels remained steady throughout so the M20 o-ring replacement that was performed this shutdown was successful (see strip of 1ACG4 - vacuum gauge for T2 systems interlock).

Note: M20 was actuated by contacting the control room, while M9 (labeled T2 blocker) was actuated through the physical ASU on the ground level of the meson hall (see picture).

For future shutdown work: the air supply valve must be kept open (tab lifted up), otherwise the solenoids won't see pressurized air. 

We ran into issues because the tab on the valve broke early into the shutdown and we didn't realize that it had to stay in the open position. 

Note: The flow on the T2 demin line went up to 1.5 gpm yesterday when we were on the blocks. The needle valve was likely nudged on accident.

Update:

1) the T2 demin flow was corrected and the air supply tab was replaced - see picture M20

2) T2 BB actuated successfully with replacement tab - vacuum remained stable

3) slow leaking valve connections were tightened again

4) water level in expansion tank corrected to just below 40 cm 

 We replaced tha O-ring on M20 at T2 and brought the target station down to vacuum, which allowed the target water pump to be turned on. 

After turning the pump on, we inspected the replaced valves for any leaks and found none - the flow on the demineralizing line was adjusted to 1.2GPM (also done at T1) by turning the needle valve.

Will observe till tomorrow to ensure that the T2 system is stable before proceeding with last system check.

 We moved M20 back to the T2 monolith and re did the connections for air + limit switches + T2 profile monitor air.

Note: T2 profile monitor electronics sparked when we moved M20 out initially. (Update: Shengli Liu from probes group performed a test on the T2 profile monitor electronics on Mar24 and found that they are working properly)

M20 started leaking when we tried to bring it to vaccum - from the monolith o ring seal and not the main shaft seal - will look to correct on monday.

Verified through measurement that the replacement o ring was the correct size (#268 0.139" diameter compared to the 0.131-0.137" that was remved - likely shrunk over use).

Update (Mar 06, 2023): M20 was lifted ~5 in above the monolith flange and we identified that the leak was caused due to damage to the o ring (image attached).

The o ring likely damaged when M20 was brought into contact wit the monolith flange, lifted up again, and brought down for a final time before vaccum check.

The above procedure was done because the flask tends to stall close to the lowermost position and prevents unlatching unless lifted systems are lowered fully.

This will be noted in updated procedures.

 T1/T2 valves were not leaky still (only a few droplets) - pending T2 cooling system pump on for final leak verification.

I have filled the water package on T2 with DI water. The expansion tank is at 38.1cm. This may change when the pump is turned on.

I have refilled the water package on T1 with DI water. The Expansion Tank is at 37cm. This may change when pump is turned on.

3x 3/4" swagelok ball valves (2x at T2 and 1x at T1) were replaced after the water cooling line was shut off at the BL1A tunnels late yesterday. 

We were able to relieve pressure from the T2 water cooler + M8 colA/B (they are the same line) at the BL1A tunnel through a drain port but no such port exists for the T1 heat exchanger line in BL1A.

The T2 heat exchanger line was virtually dry at the replaced valves after pressure was relieved.

The T1 heat exchanger line was still pressurized but while looking for a suitable port to drain at the cooling package, one of the copper fittings at the 1/4" heat exchanger line broke off at the connection to a green valve. 

~1L of water dripped from this line and this was sufficient to relieve pressure from the T1 heat exchanger line - allowing for the 3/4" ball valve to be replaced. 

Until the green valve gets replaced, the T1 heat exchanger line must remain shut off in the BL1A tunnels as the broken copper line is not plugged currently.

Update (last picture): the green valve was replaced with an equivalent 1/4" swagelok ball valve. We also incorporated two 90 deg. bends in the copper tubing to help relieve any stresses that may develop.

The M20 beam blocker was moved from the T2 monolith to the meson hall hot cell with the remote handling flask (completed ~4pm).

This operation was performed in preparation for M20 maintenance (main shaft o-ring replacement + air cylinder re-lubrication).

RPG surveyors measured 20,000 CPM from swiping the surface of the beam blocker shaft, and measured 25 mSv/hr on contact with the beam blocker.

Fields on the hot cell roof were at 100 uSv/hr at 0.5 meters from the top of the M20 BB. This should reduce drastically with the introduction of lead blanket shielding (see attached pictures). 

5 and 3 Swagelok ball valves were replaced from the T1 and T2 cooling packages respectively.

These are compression fit valves, 1/2" or 3/4" size.

1 valve in T2 was replaced previously, bringing the total # of replaced valves in T2 to 4.

Remaining valves that need to be replaced:

• T1:
  • 1x heat exchanger valve was not replaced because the heat exchanger line was not drained prior to the operation.
• T2:
  • 2x valves for the heat exchanger line were not replaced because the heat exchanger line was not drained prior to the operation.
  • 1x M8 beam blocker cooling line was not replaced for the same reasion.

Remaining valves that may need to be replaced (verify after next leak check):

• T1:
  • ~4 miscelaneous small valves (likely 3/8") were not replaced - want to check if they actually leak beforehand.
  • 2x panel mounted valves need to be inspected for leaks.
• T2:
  
  • ~3 miscelaneous small valves (likely 3/8") were not replaced - want to check if they actually leak beforehand.
  • 3x panel mounted valves need to be inspected for leaks.
  • 1x valve from the resin flask outlet has threaded connections with plumbing tape - this valve may not be leaky.

Next week, will drain all water lines associated with the cooling package before replacing remaining valves - then perform leak check. 

 (attached hand written notes)

 Update: remaining valves replaced (see: https://elog.triumf.ca/TIS/RH-Meson+Hall/320)

A 0.75" SwageLok ball valve that was taken out from the T1 cooling package was inspected (SS-45S12).

All valves in the T1/T2 cooling packages likely leaked due to damage to plastic/rubber sealing components. 

SEE PICTURES ATTACHED.

We can see that some bits of white plastic, possibly a bushing, has disintegrated and is leaking out from the valve shaft.

Additionally, there is a considerable amount of dirt/grit in and around the ball.

The surface of the valve opening exibits visible signs of wear (new valve opening is perfectly smooth).

Finally, a red-colored material is found on some small stainless steel parts which is either buildup from contaminants in the line or worn silicon/rubber lining.

Hypothesis: 90 um filter which had not been replaced caused the water in the line to build up particulates and wear plastic/rubber parts in the valve, causing them to leak. 

From the previous test (ELOG 315), we know that the leak is not due to depressurization.

Water reservoir in T1 started draining at ~7.30 am, will leave till tomorrow or overnight to fully drain. 

(Preparation to replace all leaky valves). 

Drained water samples collected for RPG. 

Maico/Eric/Albert:

The expansion tank in T1 was filled to ~36cm in the morning.

We contacted the control room to test flowrates in the T1 cooling loop after new installation of resin in the demineralizing line.

Epics interlocks tripped immediately as pump was turned on (flow too low or pressure to high) due to new resin.

Turning on the pump repeatedly extends the delay before system trips, until eventually the pump can run continuously. 

Flowrate through demin line initially too high (~2.5 gpm), adjusted flow control valve to bring it to ~1.4 gpm (setpoint at the time ~0.4-1.5 gpm).

When the pump was turned on initially, saw physical pressure gauge (dial type) bottom out/make greater than 1 full rotation (>50 Psi). 

No connection between the physical dial to EPICS so no cause for immediate concern.

All valves are more leaky than when the pumps are off - leaks definitely not due to depressurization. 

Attached: photo of the likely 'faulty' dial pressure gauge. 

It was noticed today (2023-01-17) that the valve V6, used in draining the BL1A Holding Tank to the Active Sump, was leaky. This valve should be replaced as soon as possible. It has been added to the MS Planner maintenance tasks list.

The CUNO filters were changed 2023-01-17 for:

• BL1A Holding Tank drain output, adjacent to the pump
• RH Active Sump, adjacent to the V2 valve

Both of these filters are in-line with the drainage lines from the BL1A holding tank to the active sump. It was suspected that they have not been changed in many years, if ever. Prior to changing the filters, the maximum flow rate measured from the Proteus paddle wheel sensor during draining was around 0.85 gpm (the draining procedure document indicates it should be between 0.9 to 2.5 gpm). After replacing both filters, which each showed significant clogging of sediment, the flow rate significantly increased and the maximum of 2.5 gpm was achievable. Note that when the drain pump was initially turned on, it took some time to reach the desired flow rate due to air in the water lines - the flow rate was quite low until the lines were once again filled with water.

It is suggested that these filters should be replaced every ~3-5 years, depending on usage, or otherwise as-needed (it is possible to see through the clear walls and determine how dirty they are). This MRO item will be added to the checklist of routine T1/T2 related tasks performed to prevent flow rate issues in the future during draining.

Filters were bagged and discarded as active waste.

Attached photos show old filter in bucket, and newly replaced filter, for the RH Active Sump.

The flow rate sensor used to monitor the drain rate from BL1A holding tank to active sump, located in the active sump area, was repaired. The paddle wheel, o-ring, and shaft were replaced. This part was significantly worn prior to replacement.

It is recommended to change the filter adjacent to it as it is suspected that this filter is partially clogged, resulting in low flow rate when draining the tank.

Furthermore, it is also recommended to change solenoid valve SV1 as it is suspected that it does not work.

T2 Cooling Package MRO began Jan. 16, 2023.

The following maintenance tasks have been completed for T2 (this list will be updated as work progresses):

• Drained water [2023-01-16]
• Replaced ball valve located between heat exchanger and target flow sensor FGTGT (part number SS-45S12)... this part, as well as numerous other ones, were noted to have been leaking. See e-log 308
• Replaced both CUNO filters (target water and demin. resin circuit filter) [2023-02-01]
• Replaced 90 micron demin. filter (the filter was very dirty - it was last changed in Feb. 2012 as indicated by written note) [2023-02-01]
• Inspected Hansen fittings (o-rings look OK) [2023-02-01]
• Replaced o-ring, shaft, and paddle wheel for flow sensor FGSEC (Q3). The old paddle wheel had significant wear in the shaft hole which resulted in sensor readback issues observed during 2022 operation. It is likely that this was caused by debris or a manufacturing defect which got worse over time. Photos show a comparison between old and new. [2023-02-01]
• Replaced leaky valves (see e-log 318 and 320)
• Refilled expansion tank water (see e-log 322)
• Changed M20 beam block o-ring seals, and performed vacuum check (see e-logs 319, 324, and 323)
• Tuned demin flow rate to 1.1 gpm, turned on pump [2023-03-06], confirmed system functionality (all sensors OK, no water leaks observed). Note: target water conductivity sensor (CONDTGT) took a few hours to ramp up to appropriate value. See screenshot attached of T2 state at approximately 9:15am on 2023-03-07 after the system ran overnight.

As of now, the cooling system is ready for beam. Prior to beam-on, actuation tests of key items such as the target ladder and beam blocker need to be performed. It is expected that this will be done within the next 1-2 weeks.

T1 Cooling Package MRO began Jan. 16, 2023.

The following maintenance tasks have been completed for T1 (this list will be updated as work progresses):

• Drained water
• Hansen fittings o-rings inspected: OK
• Replaced demineralizing flow metering valve (part number SS-8BK*)... this part was replaced because the equivalent part on T2 exhibited potential signs of wear during 2022 and was replaced in 2022 mini-shutdown (October). There are two things to note about this part: 1) when loosening it, it makes an audible *click* at some point during loosening rather than a smooth continuous feeling like the new replacement part, and 2) it appears to have originally been installed backwards in terms of the flow rate direction arrow. It is now installed correctly.
• Demineralizing resin can - exchanged with fresh resin (2023-01-18... dose rate of old resin which is currently located in the warm cell is approximately 40 uSv/h on contact)
• Replaced filters:
  
  • Target water CUNO filter (it was relatively clean)
  • M15 CUNO filter (it was relatively clean)
  • Demineralizing 90 micron filter (it was very dirty with tiny pieces of debris)
• Replaced leaky valves (see e-log 318 and 320)
• Refilled expansion tank water (see e-log 321)
• Tuned demin flow rate to 1.1 gpm, turned on pump [2023-03-06], confirmed system functionality (all sensors OK, no water leaks observed). See screenshot attached of T1 state at approximately 4:00pm on 2023-03-06 after the system had been running for a significant time.

As of now, the cooling system is ready for beam. Prior to beam-on, actuation tests of key items such as the target ladder and beam blocker need to be performed. It is expected that this will be done within the next 1-2 weeks.

On Jan. 13, the T1 and T2 cooling package water reservoirs were drained to the BL1A holding tank. Water samples were obtained for each system and provided to RPG for analysis.

Prior to starting draining, the BL1A holding tank water level was approximately 330L. After draining, the water level was approximately 550L (checked Jan. 16).

Upon uncovering the T1 and T2 areas to prepare for regular cooling package maintenance, it was discovered that multiple ball valves were leaking. Primarily the Swagelok SS-45S12 parts, but some SS-45S8 parts have failed as well. There were small puddles of water underneath many of the valves. It appears that the leak is due to a compromised seal. It is suspected that these parts failed from radiation exposure. They have PTFE packing material which is highly susceptible to decomposition under irradiation, and as far as we can tell they have not really been repaired/replaced (aside from a written note on one saying the seal was changed in 1997).

It is recommended to add the regular repair/replacement of these valves to the annual shutdown MRO checklist.

Once replaced, this e-log will be updated to document which ones were replaced.

It was reported on Dec. 20, 2022 that when Operations attempted to run the target ladder motor, it was unresponsive. A fault report was created: 15477.

During winter shutdown 2023, the T2 area was uncovered and the motor was inspected on 2023-01-11 by Adam Newsome and Maico Dalla Valle. The connectors and cabling appeared OK.

Upon removing the motor from its mounted position, it became clear that the coupling part between the motor and the gearbox had failed. The motor was running properly but not actually connected to the drive shaft and therefore nothing happened when ops tried to move the ladder. We confirmed that the motor runs properly in no-load condition when it is isolated from the rest of the assembly. We were able to move the rest of the drive shaft by hand and it does not appear to have any significant resistance, so it is suspected that this coupling failed due to radiation exposure over a long period of time rather than overload. The coupling part is plastic and designed intentionally to break under high loads (it is thinned out in the middle, where it snapped). Hopefully it did not actually see any high loads (i.e. there is no other root cause of this issue) and it simply degraded so much that it eventually broke.  

To resolve this issue, we will fabricate/purchase and install another one (Albert, Maico) and work with operations to test that all target ladder positions are reachable once it is installed. We will also confirm at that time that the limit switches and potentiometer readback function as expected.

It is recommended to consider the following changes to the drive system going forward to prevent this from happening in the future:

• Replace these plastic coupling parts with metal versions
• Replace the plastic coupling part that is thinned out to intentionally fail under high load with another method of system protection. The same end result could be achieved using a clutch or perhaps a fuse/circuit breaker which would effectively inhibit motion if the motor experienced overload. This would be a significant improvement over breakage because it would not require replacing the part (dose-intensive) and would be more easily recognizable.

For now, this drop-in replacement should suffice because it is not likely that the part will degrade so significantly within the next few years that this would happen again (unless there is a different root cause aside from radiation that we are not aware of!).

[update 2023-01-01] The broken coupler has been replaced by Albert Kong and Adam Newsome. The replacement part was tested - DCR operated the target ladder motor, moving it up and down through all of its positions multiple times. Everything mechanically and electrically appeared OK during motion. DCR confirmed they are able to reach each position and the proper readbacks are present - everything looks OK on their end. The issue is considered resolved and the fault is returned.

The sump level was checked by Robin Swanson on Dec. 31, 2022. It was essentially empty - see attached photo.

A fault was reported June 21, 2022 (Fault 15033) regarding T1 inlet pressure warnings... the pressure reading slowly dropped over time as seen in the attached strip tool screenshot (B1A:T1CS:PGIN). The pressure reading issue is attributed to a fault pressure transducer. The transducer was replaced on Oct. 11, 2022 my M. Dalla Valle. It is currently functioning as expected and the system is operational.

The following maintenance tasks were performed during mini-shutdown of 2022 with the goal of resolving the Fault 14966 (target water flow rate low).

• Replace the o-ring, paddle wheel, and shaft on the following flow rate sensors:
  • Q1 (FGTGT)
  • Q2
  • Q4
• Replace the demineralizing flow metering valve with a new needle valve (the old one was flaky and appeared to occasionally restrict flow)
• Re-tune the demin. flow to 1.15 gpm
• Top up the expansion tank due to water loss during the maintenance job (from 27.1 cm level to 34.8 cm)

To test the theory that the demin. flow metering valve was faulty and had an effect on the target water flow rate during operations, the valve was closed to determine if it stopped flow to the target. The result is that it did not, and only flow in the demin. circuit was affected. Therefore, it is likely that the root cause of Fault 14966 was not due to the demin. metering valve, but rather an issue with the pump, heat exchanger, pressure in the lines, or another form of obstruction. It should be noted that when the pump is turned off, all flow rate and pressure sensors drop to zero (except the pressure sensor located in the BL1A Service Tunnel) - so if the fault occurs again and symptoms show this case, it is likely that there is an issue with the pump intermittently turning off or similar.

As an aside: it is recommended in the future when changing circuit boards on the flow rate sensors to use metal screws instead of the plastic screws supplied with the parts.

Dose rates observed in the work area: 300-500 uSv/h at the target cooling package area, 1300 uSv/h at the target monolith area.

The T2-MK2 target was transported from the east hot cell to the storage pit at position 3.

As was noted in the past, position 3 is difficult to align. The flask alignment pins prevent the flask from positioning in the correct location, and it is required to have the flask just above the pins to ensure that the pins did not restrict the flask movement.

It is recommended to install a camera system to assist with the target placement and retrieval, which will help reduce the amount of dose one receives during this operation.

The target elevations measurement and beam spot checks were completed on Sept. 15, 2022. This was done after a July 2022 target exchange for target positions 3 and 5.

Updated target elevation drawing can be found in DocuShare (Collection-33652)

Mechanical Services Group changed the pre-filter in the ventilation system on Aug. 29, 2022 as per work permit C2022-08-29-2.

It is recommended that Mechanical Services change the HEPA filter as soon as possible. These filters are on order with approximately 7 month lead time. 

Edit: as of 2023-05-03, the HEPA filter has been changed.

The water level inside the BL1A active water holding tank was checked today (Aug. 23, 2022). The tank is essentially empty at this time.

On Aug. 18, 2022, Operations informed RH group that an alarm activated for the full status of the MHESA B1 RCR1 Lab holding tank. Additionally another alarm was reported in the RH control room regarding the RH sump full (Southwest Meson Hall Extension - Remote Handling area). However, there is an alarm signal for the RH sump that was seen as OK at the MHESA Lab HMI.

Upon examining the RH active sump it was clear that the water level was in fact quite high, and the red warning light on the local control box was on.

After sampling and testing by RPG the sump contents were released under work permit C2022-08-19-3 as per the procedure in Document-64834. The approximate volume of the full sump is 7270L.

When the water level was below the warning threshold (i.e. red warning light came off, and green light indicating "sump good" came on), a request was made to the control room to reset the alarm. Operations cleared the alarm at the RCR Lab controls and retired the signal defeat that was in place in the DCR (“RCRL warn” signal was defeated).

Edit (2022-08-23): BL1A holding tank water level was checked - it is empty.

It was reported today by Peter Bratt that the nuclear ventilation system for the hot cell is not functioning. At this time, the cause of the issue is unknown. Remote Handling and Electrical Services are investigating. This e-log will be updated when the issue is resolved.

Update: the issue was caused by worn belts which had been smoking. The belts must be replaced.

Some oil was noticed around the hot cell window, on the control panel, and on the ground below the control panel. A leak is suspected.

Suggest tightening all bolts to specified torque along the window, and monitoring overflow oil level (position today was marked). If the problem persists, the gasket must be replaced. Note that the gasket should likely be replaced anyway as it has not been done for some time.

The tubing for the helium wand inside the hot cell is broken and needs to be replaced. More tubing has been ordered and the job will be completed in the coming weeks.

Today myself and Adam Newsome started the leak detection steps for a target exchange and notice that the Helium line was not connected/broken from inside the hotcell.

A new line will need to be installed and the line attached to the helium wand.

This will need to be done when the target has been removed.

We will be installing a temporary line so that we can complete the target exchange first.

The temporary line will be put through the tool port as this seems the most practical and least likely to interfere with operations.

After will will have to schedule some time to install a permanent helium line with some newer flexible tubing. 

Also notice at the operator side that there is a small oil leak that appears to be coming from the hotcell window oil.

We have added an indicator(tape with pen marking) to the oil level to see how much we are loosing.

We will have to check the torques of the window bolts to ensure they are adequate.

If there is still an issue we may have to look at replacing the window seal.

Also Chad Fisher was consulted about the possible solutions 

See fault 14966: https://web.accel.triumf.ca/ncr/dbfault.pl?faultgroup=CYCLOTRON&faultno=14966&btn_submit=Getfault

The flow rate sensor for T2 target cooling was exhibiting some instantaneous drops below the warning threshold, causing nuisance warnings. It was also dropping quite close to the trip limit. The reason for this unknown and this is unexpected since the sensor was recently replaced this winter shutdown.

Until the problem can be figured out and resolved, Ops has lowered both the warn and trip threshold values to 3.0 gpm and 2.8 gpm, respectively. This was done mid-day on June 7. This has been done in the past and is deemed safe to do as it is still within normal operating parameters (and a defeat is not required). The main reason for doing this is to reduce the chance of an accidental trip if the flow rate were to drop below the previous trip threshold of 3.0 gpm. These noisy blips do not seem to be dropping below 3.0 gpm, although they are close (see attached screenshot of June 6-8 data). The sensor history will be examined weekly, and if there are no drops below 3.0 gpm, these thresholds should be adjusted back to their regular values.

Target T2-MK2 was moved from the storage location (formerly position 3) to the hot cell in preparation for replacement of end of life targets.

The T1-MK2 target was moved from the hot cell to the storage area, in position 5. Note: manual nudging of the flask was required to get it to align with guide pins properly.

The profile monitor actuation tests were completed for T1 and T2. No abnormalities were noticed in vacuum during either test.

Note: an issue was observed with T2-MK1 - see e-log 290. The issue was resolved.

An attempt was made to actuation the T2 profile monitor, however it showed up in EPICS as being not present. Furthermore, the "Remote" indicator light on the physical controller was not lit up despite the switch being in remote mode. The limit switch indicator light was working.

Upon troubleshooting with Erwin Klassen, it was determined that a jumper was missing on the profile monitor control connector (on the target end, rear of bulkhead). The jumper shorts pins A and G on the mil-spec connector on the controller end, which is for 120 VAC supply, to indicate that the device is connected. T1-MK1 was checked - this jumper was present on T1-MK1, across pins E and F of the mil-spec connector, but was not present on T2-MK1. Otherwise the connectors were identical. It is suspected that it was accidentally omitted during maintenance in the past. Regardless, a new jumper was soldered in-place on T2-MK1 across pins E and F to replicate T1-MK1. The system functioned normally after this.

Attached photo shows the completed solder job - the current state of the connector on the rear side of the target bulkhead panel.

The beam blockers for M9 and M20 were actuated, and vacuum was monitored to ensure no disruptions. Both actuation tests were successful. Motion was observed and no abnormalities noted. Cabling and air connections checked: OK.

Note: there was a safety interlock issue with M9 which prevented actuation - a relay or fuse was temporarily removed from the safety system ~10 years ago but it was not replaced. It was replaced for the purpose of this test, and then removed again.

Diagnostics group (Shengli Liu) noted a lack of connectivity between control panel and T2 Protect Monitor via the high voltage cable. Upon inspection it was noted:

• The cable was very brittle due to radiation exposure and it was cracking in multiple places
• In the SHV connector, the pins were slightly bent (bending them back into position seemed to rectify the connectivity issue).
• The cable was bent at the strain relief junction

RH Group has advised that Diagnostics replace the entire cable as soon as possible. For now it appears to be working.

The T2-MK1 target was moved from the storage area (formerly position 4) to the target station. Services were reconnected in order to prepare for normal operation.

The T2-MK2 target was moved from the target station to the storage area (position 3). Upon removal, it was measured with a pole monitor at 0.5 m to be 960 mSv/h.

 The T1/T2 water packages were drained into the B1AAWHT (280L), the water was sampled and tested by RPG, results included in elog.

Once given the all clear myself and Maico released the water from the B1AAWHT (300L). 

It was discovered that the solenoid valve SV1 which is located in the RH lab beside the sump, is broken and will need replacing, picture included in elog.

Having the solenoid broken caused the draining to go slower then normal.

The T2 cooling system was refilled with water via the expansion tank until the tank level read approximately 38 cm.

The system pump was restarted. All sensors displayed acceptable nominal values. Demin flow was tuned to 1.0 gpm.

No leaks or abnormalities observed - system running normally.

The T1 cooling system was refilled with water via the expansion tank until the tank level read approximately 38 cm.

The system pump was restarted. All sensors displayed acceptable nominal values. Demin flow was tuned to 1.0 gpm.

 No leaks or abnormalities observed - system running normally.

NOTE: the needle valve for demin flow control adjustment was behaving strangely - it needed to be tapped when loosening to increase flow in order for results to be observed. Suggest replacement next shutdown (2023) - a calendar reminder was set to do this.

All filters were replaced for the T2 cooling system: demineralizing resin circuit and target water system.

All filters were replaced for the T1 cooling system: demineralizing resin circuit, target water system, and M15.

The four o-rings on top of the T2 cooling package water reservoir were changed as per the recommended 5 year replacement schedule.

The four o-rings on top of the T1 cooling package water reservoir were changed as per the recommended 5 year replacement schedule.

The paddle wheels, o-rings, and shafts were changed at the T1 cooling package for the following sensors:

• B1A:T1:FGCOLA
• T1CS:FGTGT
• T1CS:FGDEM
• T1CS:FGSEC
• T1:FGCOLB

No major issues were noted. Some residue was observed upon cleaning the sensors. No leaks observed when turning water isolation valves back on.

The schematic which contains these sensors is TBP0902.

The paddle wheels, o-rings, and shafts were changed at the T2 cooling package for the following sensors:

• B1A:T2:FGCOLA
• T2CS:FGTGT
• T2CS:FGDEM
• T2CS:FGSEC
• T2:FGCOLB

No major issues were noted. Some residue was observed upon cleaning the sensors. No leaks observed when turning water isolation valves back on.

Two of the sensors, FGCOLA and FGTGT, experienced anomalies during the previous beam run which resulted in faults (#14656 and #14501, respectively). The system is not yet running in order to confirm whether the replacement has resolved the faults.

The schematic which contains these sensors is TBP0902.

 this Work was done on 09 Dec 2021.

Since Isaac Earle was leaving Triumf he was giving us (maico, chad, adam, and myself) a demonstration of how to remove targets, protect monitor, and wire scanner. During the removal it was noticed that one of the pins for the wire scanner was bent and it would not be possible to reattach without straightening the pins.

Luckily the fields from the ladder were low enough (374 uSv/h) and we were able to retrieve the wire scanner and manually bend the pins back. The wire scanner was reattached without incident, and now will need diagnostics to check on the target ladder to ensure all is operational.

 The RH Lab Active Sump reached the high level sensor at some point overnight before the April 8th work day.  Maico Dalla Valle noticed the alarm in the morning and notified RPG and myself.  I also received a notification by e-mail from Vicky Hannemayer who had been contacted by Ops because of the RCR alarm.

After sampling and testing by RPG the sump contents were released under work permit C2021-04-08-5 as per the procedure in Document-64834.  The approximate volume of the full sump is 7270L.

With the assistance of Maico we changed the paddle wheel, o ring, and shaft in the Collimator A flow meter (Q4) at the T2 package.

Flow meter was wiped clean to ensure o ring surface was clean.

Everything was changed without incident, and ops confirmed that the flow rate is now normal.

 March 29 Monday

- moved T1-MK1 target from beamline to hotcell  (field from target 15mSv/hr @ 0.5m)

- flushed water through position 4

- flushed air through all positions

- left air running @ 5psi overnight through position 5

- potentiometer reading "O/L" at position 3, all other positions OK

- all micro switches OK

- removed the left plug (when facing ladder) at position 1 , but after reading elogs it was found that the right plug at position 1 was having issues. Will have to replug left with new plug

March 30 Tuesday

- pumped down with cold trap, started at 10:00

- test port 2.0 E-1 Torr after 1 hour

- cold trap removed, cleared, dried, refilled

- 11:10, started pumping, found leak at lab vac tubing joint. Fixed.

- 11:16, started pump again

- 11:24, 1.0 E-3 Torr TPP, 1.4 E-9 L/s leak rate

- preformed leak check, no response at all locations

- position 4 Be target removed (required hammering). Damaged spot, potential start of hole observed at center of beam spot (see attached photos)

- position 5 graphite target removed (required hammering)

- New 12 mm Be target #108 installed at position 4

- New 12 mm Be target #109 installed at position 5

- leaving position 3 plugged as the potentiometer is reading "O/L" as indicated from the multimeter 

- New plug installed on left side of position 1

- 15:47, stated leak check pump down

- 15:57, 2.0 E-3 Torr 

leak check results

All positions were no response (NR) except

Position 1, Right side plug) 5.9 E-8 Torr

Position 3, Left side fitting) 5.3 E-9 Torr

Position 4, Left side fitting) 4.2 E-9 Torr

All locations should be H2O tight

March 31 Wednesday

- took vertical and horizontal measurements of the new position of 4 and 5 targets (updated elevations PDF sheet included)

tested profile monitor activation

- started moving at 15 psi 

- reached limit at 35 psi

- both limit switches OK

- Cleaned base of vacuum flange and top of support ring

- prepared target ready for transport with flask (ladder in position 0)

- contacted MCR to update XTPAGE 7L target details

- emailed Tony Tateyama to have PLC lookup tables updated with new target info

A trial run of remote removal of the UCN target was performed using the shielded target cask (TTA0335) developed in 2020.  The work took place between Feb 9 - 17 under work permit C2021-02-09-1.

The procedure in work instruction Document-187117 R1 was followed. Draining and purging of the water system as well as venting of helium from the target crypt was performed by Tony Hessels.  The remainder of the procedure was carried out by Keith Ng, Maico Dalla Valle, and Isaac Earle.  A maximum field of 1.7mSv/hr was measured at 0.5m from the bare target;  This is higher than the approximate 500uSv/hr field that was expected, however dose incurred during the work was minimal (Isaac: 0.05mSv, Maico: 0.03mSv, Keith: 0.03mSv).

The cask aligned well to the target arm and no modifications to the cask or UCN hardware were required to carry out the procedure.  A few updates will be required to the work instruction document based on discoveries made during the procedure; notably: 

• Cask to be installed on the west side of where the water tubing connections for the crypt flange are located;  As a result the cask lid will be installed after the cask base (with target forearm inside) is removed from the rails
• Shims are not required under blocks TTA0550 and TTA0555 in order to clear the top of the cask (the concrete and steel supports for these blocks are higher than in the UCN area shielding model)
• In addition to the six 3/8"-16 bolts for the water manifold, two shoulders screws (one on each side of the manifold) must also be removed to allow the two halves to separate

The work instruction document will be updated to include the changes above, as well as other minor updates and extra figures. 

After testing, the target forearm was reinstalled and all connections re-made.  Leak checking of the water connections and crypt flange as well as testing of the target thermocouple signals is to be arranged by the UCN Group.  The cask has been stored on top of concrete shielding on the B2 level north-east of the UCN experimental area.

Various photos taken during the procedure are attached.

 The following work on the T2-MK1 target was completed sporadically over the last few months:

Sept 22:
- T2-MK1 target moved from storage pit to hot cell (new camera view to check flask doors and latch engagement works great)
- Rubber pad from storage pit standoffs was stuck under flange - removed and cleaned base of flange
- Preliminary leak check: target positions and  delta seals no response, left bellows (facing ladder) had tiny response up to 3.0E-8 with recovery in ~5s, right bellows no response
- Removed plugs at Position #1 and #3

Oct 14:
- Installed 12mm Be target #106 at Position 1 (confirmed with Yuri that it may be useful to have a short Be target on the T2 ladder)
- Installed 5cm target #301 at Position 3
- Leak checked target positions: no response, only mild delayed drift
- Position 4 plugs removed

Nov 10:
- Installed 5cm target #302 at Position 4
- Repeated leak check: no response at all target positions
- Took vertical and horizontal measurements for all new targets
- Position limit switches and potentiometer confirmed working ok

Nov 24:
- Moved to Storage Pit Hole #4
 

The T2 Collimator A flow meter signal (B1A:T2:FGCOLA) began fluctuating and occasionally reading 0gpm on Sunday Oct 25 causing BL1A trips and machine down time.

On the maintenance day today (Tues Oct 27), the T2 package was uncovered and the T2 Col. A flow meter was opened.  No unusual deposits were observed as seen recently with other flowmeters for BL1A magnets on the CuALCW system.  Some minor wear was observed on the dowel pin shaft, and the paddle wheel didn't spin quite as freely as when new.  A small amount of reddish/brown film (rust?) was seen on the internal SS faces.  The faces were wiped clean and new dowel pin, paddle wheel and o-ring were installed.

The pin, paddlewheel, and o-ring were also replaced for B1A:T2:FGCOLB (no wear observed on old pin) and B1A:T2CS:FGTGT (some wear observed on pin, more than Col. A).  They were not replaced on B1A:T2CS:FGSEC or B1A:T2CS:FGDEM at this time, as neither of these has a noisy signal, and neither can cause trips in the control system.

After all paddlewheels were re-installed and the system restarted, the flowmeters behaved as expected.  The fluctuations that occurred on Sunday were not happening this morning prior to the change, so we're unsure at this point if this has solved the problem.  StripTool was used before and after the change to observe fluctuations in the readbacks, and no noticeable difference was observed.

During testing of the flow meters we observed that B1A:T2:SVCOLB seemed to not respond one of the times it was commanded to close, and one or two other times it seemed to close after a few seconds delay.

Note: All T1 and T2 paddlewheels, pins, and o-rings were changed earlier this year.  They're scheduled to be replaced every 2 years as per the T1/T2 maintenance plan.

The following work was completed during the 2020 shutdown period for BL1A (Jan-Aug 2020):

• T1 and T2 EPICS data checked for anomalies.  None found, all sensors behaving as expected
• T1 and T2 cooling package water drained to storage tank, sampled, and released
• All cuno filters changed (3 for T1, 2 for T2)
• All T1 and T2 proteus paddle wheels changed
• T1 and T2 hansen fitting connections inspected: all good, no leaks
• Both packages refilled, restarted, demin flow tuned to 1gpm, and checked for leaks: very small leak from 3/4" line going into heat exchanger on T1 target loop (not new, tightening doesn't help) 

• Profile monitor air hoses inspected: ok
• Air amplifiers inspected: cycle time ok, no leaks
• M9 beam blocker overhaul completed including new air cylinders
• M9 and M20 BB actuation checked
• T1 and T2 profile monitor actuation checked
• T1 and T2 ladder movement checked
• XTPAGE 7L updated to indicate which targets spent
• Target ladder status sheet updated
• ASCO valves for M9BB, M20BB, T1 prof mon, T2 prof mon all serviced (with rebuild kit)

August 25, 2020 update:

- The cap block was replaced over T2
- Actuation of the M9-T2 and M20 beam blockers was tested.  Both moved fine and limits displayed correctly

 Mike Vogel reported that alignment of the Q1Q2 assembly w.r.t. the T2 and B1 flanges was completed on July 7.  This was performed in the test stand with the flange positions set to the measurements taken in the 2019 shutdown.  Alignment to the theoretical location was achieved within 0.1mm.

The axis of the Q1 and Q2 magnets were aligned to each other previously within similar accuracy.

The diagram below shows the wiring for the two connectors on the service stand (the wires running from the stand to the power supplies.  (see TB23400 for details)

 The M9 Q1Q2 magnet assembly lift beam has been installed according to drawing TB23401.  The side-to-side as well as beam-axis direction adjustment features were tuned so that the assembly is level when lifted.

Aug 5, 2020 update:  All fasteners on the lift beam assembly have been torqued according to the specification on drawing TB23401

The Q1Q2 water header (TB23481) was mounted this morning as per installation drawing TB23400.

Heat shrink tubing was applied to the B1 power cables (two positive, two negative) over the length of cable in the highest field which was showing some signs of radiation damage on the insulation.  The tubing used was McMaster-Carr PN 75665K932 (Clear PTFE, 4:1 shrink ratio, 1.0" ID before shrinking), as recommended by Syd Kreitzman.  Each piece of tubing is 4' long, and two were used per cable.  The first one was shrunk along its full length, then the second piece was only shrunk where it overlaps the first and the remainder was left loose.  This was done to minimize time required in the high dose area.

Hand-stacked concrete bricks were removed from the location where the new piping will run to the water header.  This is scheduled to take place between July 6-8

 Laser tracker scan data from 2017 and 2019 attached as a backup.  Original files from Beamlines Group who also keep copies on \\trwindata\experimental support\alignment\T2-M9_data

 The 10-pin thermal switch wire, 2-pin steering wire for Q2, and a 1/4" copper tube for remote helium leak check were installed through the service chase with the power cables and connected to the M9 service stand.  Approximately 10m of extra cable (2-pin and 10-pin) was coiled and left in the cooler section of the service chase after the first sharp bend.  The cables and the tubing are currently run as far as the vertical cable tray "waterfall" located near the M11 area.

The M9 beam blocker was returned to the beamline today using the RH shielded transport flask.  The top flange o-ring was replaced before installation.

After installation the "Lift T" assembly was removed, and electrical cable and air supply connected.  Correct functionality of the limit switches was confirmed with the control room.  Actuation of the beam blocker could not be tested at this time because control takes place through the M9 area ASU.

M9 beam blocker MRO work was completed in the hot cell today.  This included replacement of the main shaft seals and replacement of the air cylinders.  Actuation was tested after replacement and the blocker moved smoothly through the full length of travel with approximately 45psi air pressure.  It is ready for transport back to the beamline.

 The M9 Q1Q2 "foot" plate (TB23363) has been installed over the floor plate.  A nominal gap between the plates of 1/2" was set before lowering the plate, it was then leveled using the three adjustment bolts resulting in a larger gap.  After leveling, the plate was lifted to remove the spirit levels and lock the position of the leveling bolts.  The plate was then re-lowered and the tiedown bolt assemblies (TB23383) were installed using 50ft*lbf of torque (about the maximum practical amount with standard 20' pole tools)

 A 4” Marmon aluminum blank-off flange and o-ring were replaced on the south port of the T1 target monolith vacuum vessel. This port was accessed from the 1A Service Tunnel were the repair took place.

See the attached pdf file “1AT1 Vacuum Leak Repair-Feb2020” for a summary of the repair.

 The M9 service stand floor plate (TB23351) was installed in the M9 pit yesterday.  Prior to installation all 5 threaded inserts were tested using a 1/2"-13 stud tool on an RH pole.  The west most hole (shown in red below) had damaged threads and the stud could only be threaded in 2 turns.  All other locations (shown in green) had usable threads with 6-8 full turns possible with the stud tool.  The plate was installed using RH floor plate bolts (TB23382) at all four locations with viable threads.  The bolts were torqued to 30 ft*lbs which on a bench test was shown to fully compress the belleville washer.  The standard torque spec for 1/2"-13 stainless bolts (43 ft*lbs) was not used to allow easier disassembly in the future.

 The T1-MK2 target was moved from the hot cell to Hole #5 in the storage pit.  The M9 beam blocker was then moved from the beamline to the hot cell for MRO work.  A field of 14.4mSv/hr was measured at 0.5m from the beam blocker during removal.

 Annual MRO work on the T1 and T2 cooling packages is now complete.  The following tasks were performed:

• Water from both packages drained to the South TNF holding tank (samples taken)
• All Proteus paddle wheels, shafts, and o-rings replaced (5 each at T1 and T2)
• All Cuno filters changed
• Archived data for all sensor readbacks inspected.  Only faulty sensor was B1A:T1CS:FGDEM
• Both packages refilled and restarted; Operating smoothly; Inspected for leaks, none found; Demin flow at ~1.0gpm
• B1A:T1CS:FGDEM readback is fixed with new paddle wheel
• South TNF holding tank water released after approval from RPG
• Old filters added to plastic pail with other T1/T2 filters located directly north of the BL1A blocks boot-box area
• Active water from filter canisters dumped into the RH lab active sink

The following work was done to the T1-MK2 target in the hotcell between the dates of May 24, 2019 - Jan 30, 2020 (See E-Log #251 for further information):

• May 24: Flushed position 2 with water, purged all positions with air.  Left air running through position 0 over the weekend
• May 27: Switched to position 5, left air running to dry out bellows
• May 28: Started pump out with cold-trap;  Reached "fine test" (TP pressure < 10E-2 Torr) after ~10-15 minutes, left for 2 hours, removed ice build-up then restarted leak detector;  
  Down to 1.0E-3 Torr TP pressure and 1.0E-8 Torr*L/s after ~10min
  Leak checked with 5psi helium, ~0.5s spray: Response at lower delta seal right side (when facing ladder) up to 3.8E-6 Torr*L/s and drift responses from nearby locations
• June 27: Position 1 target removed - no issues, required light tapping with hamme
  Position 4 target removed - lots of hammer hits required
  Both target beam spots photographed.  Saw what looks like hole on Position 4 entrance window (see attached photos)
• July 8: Protect monitor inspected visually using Nikon level by Dave Cameron and Shengli Liu - Nothing abnormal observed
  Electrical check w/ multimeter: all OK
  Observed ground to HV plates while moving ladder from Position 5 to 0 - all OK
  Observed ground to HV plates while moving profile monitor IN then OUT - all OK
  Found intermittent short from HV to ground when shaking the target assembly or tapping the protect monitor cassette:  must replace
• July 18: Cleaned Position 1 and 4 Swagelok sealing surfaces on the target ladder using the Dremel tool with burgandy Scotchbrite
  Ready for new targets to be installed
  Plugs installed at Position 4
• July 24: Target #105 installed at Position 1
  Started leak detector pumpdown with cold trap;  Only pumped down to 4.0E-3 Torr;  Big helium response at lower right delta seal (10E-5 Torr*L/s range)
  Concluded that the leak got worse during the target replacement (likely from the hammering required to remove Position 4 target
  Tightened bolts for lower right delta seal then repeated leak check: same results, therefore must replace seals
  Received replacement protect monitor cassette from Probes Group and installed
• July 29: Probes group did blip test on new protect monitor - all OK
  We repeated the test for shorts with shaking and tapping on the target assembly and monitor - no shorts observed
  Concluded that protect monitor issues is now fixed
• Early Jan 2020: Lower cone and plate removed, ladder removed (old style w/o plate at top)
  Removed seal spacers, the one for the lower right delta seal was stuck, had to use a flat edge screwdriver as a wedge with aluminum hammer to dislodge
  Removed delta seals, observed some deposits or corrosion at lower right seal location
  Cleaned spacers using Scotchbrite on Dremel
  Used 3D printed jigs with glued on burgandy Scotchbrite to clean sealing surfaces (no air ratchet or dremel, polished with manipulators until a clean shiny surface was produced)
  Had to remove bolts for the ladder flanges to clean the surfaces, and when reinstalling one of them it cross threaded and couldnt be removed
  Bolted instead at different hole locations (still at opposite sides of flange) but also had to switch bolt locations on upper seals to do this because bolts will interfere if using same threaded hole
  Eventually got everything back together with new delta seals installed (TBP0454B, machine shop order #44138)
  Note: the upper delta seals were not disturbed - new bolts installed and torqued before old ones were removed
• Jan 22: Leak check, pumped with cold trap: down to 0.0E-10 Torr*L/s and 1.0E-3Torr TP pressure after 9.5 minutes
  Using 5 psi helium, 0.5s spray;  No response at any previous locations;  Very small drift response up to 0.5E-10 Torr*L/s
  Concluded that leak is fixed
• Jan 26: Base plate and cone reinstalled - no issues
• Jan 30: David Kyle and Shengli Liu re-tested the profile monitor and protect monitors - No issues
  Measured vertical and horizontal positions for all targets and protect monitor, updated measurements sheet (PDF attached)
  Ladder moved to Position 0
  Target assembly ready to be moved to storage pit
   

 The second (and final) 55 gallon drum containing active items from the Meson Hall East Hot Cell was transported today from the Warm Cell to the Cyclotron East Vault Tunnel.   The updated form is attached.

The following items were moved today from the Warm Cell to the Rad Waste Pit for long term storage:

1. A 55 gallon drum containing various active components from the East Hot Cell.  The drum was labelled on the body and lid with the identification number RWC-55-01-2019.

2. The old M9 Q1Q2 service stand, which was removed from the beamline in the 2019 winter shutdown

Photos taken during the move are attached, as well as a copy of the loading record form for the drum.  The form has also been given to RPG.

A second drum (RWC-55-02-2019) remains in the Warm Cell, and will later be moved to the cyclotron vault tunnel for storage.  It was resurveyed after the first drum was removed and deemed too low-level for storage in the pit (300uSv/hr at 0.5m)

The following work was completed on both the T1 and T2 target alignment frames

• One of the short taper pins on the top of the frame was swapped for a longer version to simplify flask alignment to the frame (now one pin can be aligned at a time instead of both concurrently)
• Three permanently mounted cameras installed to view east/west alignment, north/south alignment, as well as hook, latch, and door position
• LED strip lights installed on the bottom of the frame for better illumination during alignment of target devices
• All video and power signals routed through a single connector (9-pin CPC) for ease of use
• Frame painted with blue Rustoleum metal paint
• Hole positions, alignment orientation, camera position, etc labeled on frame

After extensive testing over the last 2 weeks it was determined that the protect monitor on the T1-MK2 target had an issue with intermittent shorting between the high voltage line and ground (movement of the target ladder was causing the short to come and go).

On the May 21 maintenance day the T1-MK2 target was moved from the beamline to the hot cell (102mSv/hr @ 0.5m).  The T1-MK1 target was then moved from Storage Pit location #3 to the beam line (as in the past, alignment of the flask to the #3 position hole was challenging, and the flask had to be suspended above the alignment pins to achieve alignment with the hole - In the future this storage pit location should be avoided).

Cameras were mounted on the T1 alignment frame for remote viewing of alignment with the hole as well as flask door position and hook latch position.  The camera locations chosen provided useful views and did not interfere with the target or frame.  After procurement of additional mounting brackets the cameras will be permanently installed at these locations (already completed on the T2 frame).

The target moves were completed successfully, vacuum pump-down was normal, and electrical checks of the T1-MK1 profile monitor and protect monitor were normal.  When start-up of the cooling system was attempted problems were encountered with the control system.  Multiple channels on the digital output module were shorting, causing the power supply fuse in the PLC cabinet to blow.  This was eventually tracked down to two damaged cables (B1A:T1CS:SVSEC and B1A:T1:SVCOL) which somehow had been crushed during removal of shielding.  The cables were replaced and all cables secured as much as possible in the vertical chase leading up from T1.  After replacing the cables we discovered that the digital output for the SVSEC signal was stuck in 'drive' status.  The module was replaced and then the control system was working normally (approximately 11am on May 23rd).

The T1 cooling system was started (no change to 1ACG2 vacuum), ladder movement through all target positions was tested, and profile monitor actuation was tested.  Updated target info sheets were submitted to Ops (attached in PDF format) and XTPAGE 7L was updated.

Diagnosis and repair of the T1-MK2 target will take place in the hot cell over the coming weeks.

Cyclotron Fault #12266 was returned.

 Completed work at m9 beam line:

- Removed existing service stand and associated cable and water services. Legacy b1 water services still need to be removed for future magnet installation. Old service stand is currently in warm cell waiting for disposal.

- M9 Q345 and vacuum roughing line were cleaned by CMMS group.

- M9 B1 is disconnected but remains in beam line ready for removal and refurbishment for 2020 shut down. Relevant ports have been sealed off.

- New Q1 Q2 baseplate was test fit and did not fit with guide post bolted in place, interferes with T2 port blank off plate.

- 3d lidar scans of magnet space were done.

- Leica tracker data was taken of the holes in the poured in place concrete pad.


Photos from shutdown are on remote handling group folder, under "\\trwindata\groups\remote handling\Photos\2019\2019 m9 project"

The following work was completed between Jan 16 - Feb 19, 2019:

• T1 and T2 cooling packages drained to South TNF holding tank, water sampled and tested, released after approval from RPG
• T1 resin can swapped
• All Cuno filters changed at T1 (3 total)
• T1 inlet (T1CSPGIN) and outlet (T1CSPGOUT) pressure gauges replaced (archived data on EPICS indicated both starting to fail)
• T2 target water Cuno filter replaced
• All T1 and T2 solenoid valves changed to new models without power management circuit which will hopefully solve the problem of frequent failures (SASP0670)
  Parker brass body valve 73218BN4UN00 with solenoid C222C2 solenoid used for T1CSSVSEC and T2CSSVSEC
  STC stainless steel body valve 2S160-1/2-2-D with solenoid 2W200C-2-D used for T1CSSVTGT, T1SVCOL, T2CSSVTGT, T2SVCOLA, T2SVCOLB
  Necessary changes to PLC BOP wiring made by Tony Tateyama
  All new valves confirmed operational and leak tight
  Cyclotron Fault #11955 returned
• T1 and T2 profile monitor air cyclinder flow control valves replaced with new models tuned to ~1/8 turn open.  Part is Rego F125B (see PO #3044305)
• T1-MK2 profile monitor air cylinders replaced (both were leaking through the top seal)
• T1 and T2 profile monitor actuation checked - both move in and out slowly and smoothly
• M20 BB actuation checked - movement is slow and smooth
• T1 and T2 ladder movement checked - both ok
• T1-MK2 male Hansen fittings replaced as well as female Hansens which connect to them (these are the last remaining Hansen fittings to be replaced at T1/T2)
• Both cooling packages filled and started, now running smoothly.  Inspected for leaks (none found).  No change to 1A vacuum when packages started
   

Note 1: The T1 package water pump is slightly noisy when running (it has been this way since the replacement pump was installed last year) - Will investigate further at a later date

Note 2: In the week of Feb 11-15 Vacuum Group pressurized the T1 target cooling circuit with 30psi helium to try to find the vacuum load on the T1 volume.  This was done twice with the beamline vented for a few hours in between in case water had frozen at a leak location. The conclusion after testing is that the T1 cooling water is not the source of the vacuum load at T1.  Helium was also sprayed all around the top of the T1 monolith (T1 target and M15 permanent magnet) - there was no response on the leak detector

My total dose for this work was 0.15mSv

2018 Cyclotron lid up problem. Station #9 was  found to  have the motor wires along the beam plain had damaged wire insulation. There were bare wires in the conduit resulting in the wires shorting out. The wires were repaired with a quick short replacement. Lid went up for in tank work. After lid down the problem was investigated. We checked the radiation levels   at all  of the stations to come up with a plan to determine witch of the stations had the highest levels. We found that stations 1,3,5,7,9, and 11 had the highest levels. Station 1 was inspected and all wires were replaced to the T elbow. Station 3 all wires were replace to the T elbow. Station 5 all wires were replaced  to the T elbow. Station 7 all wires were inspected and were in good condition and put back into place. Station 9 all wires including syncro were replaced all the way to the junction box. Station 11 motor wires were good, but the jog wires and the syncro were replaced to the pull box. All of the jog switches on stations 1,3,5,7,9,and 11 were falling apart when wires were inspected.  New switches were ordered and assembled. At the start of 2019 shutdown all jog switches were replaced on stations 1,3,5,7,9, and 11. After all the jack work was finished the jog switches were check and were working well.

The T1 and T2 cooling packages were inspected today while running.  No water leaks were observed at either package including the high-active loops and copper active. 

The T1 pump was fairly noisy (video was taken to record the sound).  This is the new pump installed in August 2018 (see E-Log #242).  Maico reports that it had a similar noise when started after installation in August, and they were unsure if it was a normal sound or not.  The pump it replaced ran quietly, as well as the pump currently running at T2.  Further investigation will be performed this shutdown to try and determine the cause of the noise and appropriate action.

Both packages have been turned off and pump contactor control signals disconnected in preparation for MRO work and power outage this weekend.

Randomly checked on the nuclear ventilation for the Meson Hall Hotcells and discovered they were not running again (both magnahelic gauges displayed zero). Attempted to restart the fan manually and the control box seems to not be working. Matt McClean has been contacted and they will look into the problem.

The nuclear ventilation is currently not running in the RHMH hot cells.

On 23rd October we restarted the T1 and T2 water packages after the mini shutdown and after the site power outage. The rotary collimator was reporting no water flow and it was assumed the valve coil (B1A:T1:SVCOL) had failed again. T1 water package was uncovered the next day (24th) and we went to change the coil. Water flow was returned to the collimator and T1 water package was re-covered.

During the replacement, one of the nearby pressure sensor wires accidentally made contact with a nearby paddle wheel sensor shorting out the output connectors and making the flow read between 8 to 1999 GPM. The offending wire was observed and removed after going back down to have a closer look.

It is not believed the coil failed from the site power outage on the 19th. The water packages had been left off since the beginning of the mini shutdown (2nd of October?).

 On Friday Aug 31 at approximately 3am T1CS:FGSEC dropped to 0gpm flow.  This was confirmed to be a real reading when T1CS temperatures continued to rise steadily.  Beam was turned off, and T1CS later turned off by operators (temps continued to rise due to heat from pump).  The cooling package was uncovered in the morning, and the T1CS:SVSEC solenoid was replaced by Doug Preddy and Keith Ng.  This solved the problem and the package was restarted and then covered.

This is the latest of several failures of these new solenoids at T1/T2 since installation in the 2016 winter shutdown.  It is suspected that the "enhanced electronics" in this model of the valves is vulnerable to damage from radiation.  ASCO does not carry 24V DC valves in their "General Service" line which does not have the enhanced electronics.  Valves from other suppliers are being investigated.  If a suitable 24V DC model cannot be found then we can revert to the 110V AC ASCO valves which we know are reliable.  This will require relays and wiring to be done by electricians.

 The water in the South TNF Holding Tank was released today under work permit #C2018-08-28-3.  The tank contained approximately 100L from the T1 cooling package (drained Aug 16, 2018) and 350L from the 500MeV system (drained April 23, 2018 by Steve Sapriken).

On Wednesday August 15th the T1 cooling package tripped off, and could not be restarted.  On the EPICS screen the pump device went into an immediate time-out each time we attempted to turn it on.  An electrician was involved in the debugging, and issues were found with the 480V supply.  The breaker switch in the 480V supply panel was found to be damaged, and was replaced.  The relay in the contactor box for the pump motor thermal protection was also replaced.  These new parts did not solve the problem.  Resistance measurements between the phases of the pump motor revealed ~10ohms between phases A-B, but ~250ohms from B-C and A-C (measured from inside the contactor box).  The T2 pump and a spare replacement pump were also tested, and had ~10ohms between all phases.  It was therefore concluded that the pump motor, or possibly the wiring between the contactor and motor had failed.  The T1 cooling package was uncovered and drained by the end of the day.

On August 16th the old pump was disconnected and de-wired.  Damaged wire insulation on the pump motor was found, which likely caused a short, leading to failure of the pump motor, and cascading failures of the electrical system and possibly also the control PLC.  A spare pump (Chempump GB-3K-1S) was installed with custom cut gaskets.  The pump was re-wired by an electrician.  The cooling system was re-filled, and start-up was attempted, however there were remaining issues with the electrical system.  By the end of the day various testing by Controls Group and Electrical Group determined that the pump motor contactor also had failed and required replacing.

On August 17th the contactor and over-current protector were replaced by an electrician.  When the breaker was switched on the pump unexpectedly started immediately.  This was found to be caused by the control system drive signal module being faulty causing the output to be stuck on.  Failure of this module may have been caused by excessive current draw due to the damaged contactor.  The failed module was replaced, and the PLC and IOC were both restarted.  After this the system worked normally.  The pump was inspected by Maico Dalle Valle, no leaks were observed and operation seemed normal.  Shielding above the cooling package was replaced, and BL1A was restarted.  Cyclotron Fault #11527 was returned.

A new spare contactor and over-current protector have been ordered (Allied Electronics LC1D09BD and LRD12, Requisition #1037940).  These will be labeled and given to the Electrical Group.  A new spare T1 / T2 pump has also been ordered (Chempump GB-3K-1S, Requisition #1038041) which will be stored in the RH Meson Hall Hot Cell Lab Tool Port Boot Box Area.

A data sheet and quote for the replacement pump are attached.

On Saturday July 21st the T1 Collimator solenoid valve would not re-open after tripping closed due to a (planned) loss of vacuum.  Shielding was removed and the valve solenoid was replaced which fixed the problem.  The original valve body was left in place.

This is the second time the solenoid at this location failed (See E-Log #239), and the latest of several failures of these new 24VDC solenoids at the T1 and T2 cooling packages.  Clearly they are not reliable in this radiation environment, so I will investigate an alternative to be installed in the 2018 Fall Shutdown or 2019 Winter Shutdown.

Cyclotron E-Fault #11420 has been returned.

 The T1 cooling package outlet water pressure sensor (B1A:T1CS:PGOUT) was replaced this morning with a new unit.  Over the last few months the sensor readout had been steadily decreasing (from around 20psi to 4psi) with no corresponding decrease in inlet pressure, and no change in the circuit flowrate (this is how these pressure sensors typically fail).

After installing the new sensor the readback in EPICS returned to the expected value.  Although there are no alarms or trips associated with this signal, it is still useful as a diagnostic tool.

Flow stopped through the T1 collimator cooling loop around the same time as a power bump this afternoon.  The problem was narrowed down to failed solenoid valve B1A:T1:SVCOL by comparing coil resistance to other units without issue (~1.0ohm vs 3.6ohm for working units), measured at the T1/T2 PLC cabinet.  Shielding above the T1 package was removed and the solenoid coil was replaced with a new unit.  After replacement the valve works as expected and B1A:T1:FGCOL reading is normal.  I inspected the T1 package while it was uncovered: no leaks observed.

During first injection of beam down 1A problems were encountered with the T1 protect monitor.  Testing was done by Probes Group from the mezzanine leading to the following investigation and repair work.

May 9th:

- T1 area uncovered, Probes Group performed testing at the top of the monolith confirming that the problem was below the vacuum flange
- T1-MK2 target moved from the beamline to the hot cell.  Measured field 800uSv/hr at ~0.5m
- Visual inspection of protect monitor performed:  No obvious issues with monitor plates.  Some small impact holes visible on exit side of monitor, similar to damage on the previous monitor from Oct 2015
- Electrical check performed on old protect monitor in hot cell, confirming the previous results
- Failed protect monitor removed, resistor pack installed and tested with expected results
- Profile monitor insulation found to be blackened and cracking, so the profile monitor cassette was also removed
- New protect monitor and profile monitors installed
- Electrical check performed on both monitors at the hot cell: Results good for both

May 10th:

- Vertical and horizontal positions measured for the new protect and profile monitors
- Profile monitor actuation tested:  started moving ~10psi, fully actuated ~50psi, monitor limit switches ok
- Measured total resistance of potentiometer: 20.26kOhms
- Target windows inspected and photos taken (stored on Isaac's hard drive), nothing unusual encountered
- T1-MK2 target returned to beamline, vacuum pumped down without issue
- T1 cooling package started fine with no change in vacuum
- T1 profile monitor actuated in and out.  Movement smooth and slow
- Electrical check done on protect and profile monitors from mezzanine: both good

The M20 beam blocker main shaft seals were replaced as per the T1/T2 target device maintenance schedule.  Both air cylinders were found to be leaking past the seals, and the blocker would not actuate in the hot cell even with 100psi (max site air) so they were also replaced.  With the new cylinders the blocker actuated smoothly at about 45psi.  It is recommended that the air cylinders be replaced every second service (once per 10 years).  The blocker was returned to the beam line and smooth actuation and limit switch functionality were confirmed.

The following final checks were performed on T1/T2 devices:

- Lower air amplifier started and tuned to 120psi output
- Air amplifier cycle time measured:  70 seconds for two cylinder turn-overs (1 complete cycle)
- T1 and T2 profile monitor actuation checked:  T1 reached in and out limits but speed unknown because target covered;  T2 traveled smoothly in and out and reached limits as expected
- T1 and T2 target ladders were moved to the various targets that could be used this running period (#1 for T1;  #1, 3, and 4 for T2) - No issues

The T1 and T2 targets are ready for the running period.  Shielding can now be replaced over T2.
 

 The following work was performed on the T1 and T2 cooling packages during the 2018 shutdown:

- T1 and T2 cooling packages both sampled and drained to South TNF holding tank
- Water released after testing and approval from RPG

- T1 high active and M15 filters changed (demin. filter to be changed next year)
- Replaced all T1 Proteus paddle wheels, shafts, and o-rings
- Replaced T1 male and female #6 SS Hansen fittings at the package/hose interface
- Replaced T1 #4 female Hansen fittings for the resin can (now all Hansen fittings on the high active loop have been replaced)
- T1 package filled and started, no change to T1 area vacuum
- Demin flow tuned to ~1.0gpm
- Package inspected while running, no leaks observed

- T2 high active filter changed (demin. filter to be changed next year)
- Replaced all T2 Proteus paddle wheels, shafts, and o-rings
- Replaced T2 male and female #6 SS Hansen fittings at the package/hose interface
- Replaced T2 #4 female Hansen fittings for the resin can (now all Hansen fittings on the high active loop have been replaced)
- T2 resin can changed;  old resin can flushed with water then air in Warm Cell and left to decay
- T2 package filled and started, no change to T2 area vacuum
- Demin flow tuned to ~1.0gpm
- Package inspected while running, no leaks observed

The cooling packages are functioning normally and are ready for the running period.

The following is a summary of MRO work done on the T2-MK2 target during the 2018 shutdown:

Jan 8:
- Target moved from beamline to hot cell;  130mSv/hr at 0.5m

Jan 10:
- Position 4 (12mm Be) beam spots photographed.  Spot is slightly high and left (from beam's eye view), also smaller and rounder (less eliptical) than usual - See E-Log #232

Jan 11:
- Flushed position 0,1,2,3,4 with water, then 4,2,1,0 with air
- Left with 10psi air running overnight in Position 0

Jan 12:
- Moved to position 4, 10psi air running through over the weekend to dry cooling lines

Jan 15:
- Pumped out water lines with leak detector
- Frist pump down blanked off tubing and new vac hose (no LN2): ~15 mins to get to 0.0E-10 TorrL/s
- Pumped out target w/ cold trap:  ~7 mins to get to 0.0E-10 TorrL/s (left pumping for 2 hours)

April 2:
- Removed plugs at position 3 and 5
- Pressure tested, leak tested, and recorded alignment dimensions for first batch of new 5cm targets (#301, 307, and 309) - Records stored with targets in RH lab office
- Installed new 5cm beryllium target #309 at Position 3, and target #307 at Position 5 
- Helium leak check on target water system:  Baseline 0.0E-10 TorrL/s leak rate, 1.0E-3Torr, half second helium applications @ 4psi
      Positions 1, 2, 3: no response
      Positions 4, 5: up to 5.0E-10 after delay (appears to be drift)
      Lower 'C' seals: no response
      Upper-left 'C' seal: up to 1.2E-9
      Upper-right 'C' seal: up to 2.4E-9
      Left bellows: up to 6.0E-9
      Right bellows: up to 3.0E-9
- Deemed water leak tight and ready for use (based on previous leak check results and operating experience)
- Tested all target position micro switches: all OK
- Tested profile monitor limits: both OK
- Tested profile monitor actuation: off down limit @ 5psi, on up limit at 35psi
- Tested potentiometer: slightly noisy signal from position 0 to 2 (no change from before)
- Measured Position 3 and 5 elevations

April 3:
- Measured Position 3 and 5 horizontal alignment wrt ladder
- Quick repeat of leak check:  no change

April 6:
- Target returned to beamline
- Vacuum pulled on T2 area (1ACG4 down to 5mTorr in ~2.5hrs, was 4mTorr before venting yesterday)
- T2 cooling package started with target at Position 0 - no change to T2 vacuum
- Target information sheet and elevations data delivered to Ops
- XTPAGE 7L information updated
 

 Five 5 gallon metal pails with active waste were moved from the Warm Cell to the Cyclotron East Vault Tunnel today for long term storage.  The pails were labeled with radioactive item tags and storage item numbers (1442 through 1446).  The pails contain used resin from T1, T2, and STF, as well as one pail with active metal parts.

 The RH Lab active sump level reached the high limit yesterday due to an overflow of the ultrasonic cleaner in the Rad Waste Area.  The sump was sampled by RPG, and after analysis the water was approved for release.  The water was pumped out today.  Local and MCR alarms functioned as expected.

 The T2-MK2 target was moved from the beamline to the hot cell on January 8th.  The field at 0.5m was 130mSv/hr.  Photographs of the beam spot of the Position 4 target are attached below.  This is a 12mm beryllium target (ID: "JUL.06#4).  This is the only target on the T2 ladder that has seen beam since it was installed on October 10, 2017.  A shielding plug was installed on the T2 flange.  A new 5cm target will be installed on the T2-MK2 ladder after fabrication is complete.  Regular testing and maintenance of the target assembly will also be performed before it is returned to the beamline.

The M20 beam blocker was also moved from the beamline to Hole #7 in the storage pit.  The field at 0.5m was 22mSv/hr.  A blank-off plate was installed on the M20BB flange (no o-ring).  Seal replacement and general inspection/testing will be performed before it is returned to the beamline.

Six 5 gallon pails were transferred this morning from a temporary storage bunker in the Warm Cell to the Rad Waste Pit for long term storage.  The pails contain various BL1A active waste from the East Hot Cell including dozens of irradiated beryllium and graphite targets.  The field from the pails ranged from 0.1 to 9mSv/hr at 0.5m.  The work was performed according to the clean-out procedure (Document-142965).  Photographs and records of the pail contents, field, weight, etc have been uploaded to Docushare Collection #20185 and hard paper copies were given to RPG.

All remaining waste in the east hot cell is too large to fit into 5 gallon pails, so it must be packaged in a 55 gallon drum.  Some long items will need to be cut in the cell before hand.

The field on the hot cell roof is now less than 5uSv/hr in working areas, and a maximum of 55uSv/hr at 0.5m above the east hot cell opening.  The field was previously > 50uSv/hr in some working areas, and > 200uSv/hr above the opening.

Before and after photos of the hot cell interior are included below:

 Around 10pm on Sunday Oct 22 beam was tripped multiple times due to low flow for the T1 collimator cooling circuit (B1A:T1:FGCOL).  Temperature of the collimator (TC7COL, TC8COL) was monitored with reduced beam current and no increase was observed leading to the conclusion that flow was OK and the flow gauge was faulty.  Normal operation continued until around 1:30am on Oct 26 when the collimator thermocouples both rose quickly, causing the beam to trip when they reached 60C.  At this point we concluded that there was no longer flow in the cooling circuit.  1A was defined off and shielding was removed down to the T1 cooling package.  The problem was found to be the solenoid valve which would open when actuated, but close in less than a second.  The flow meter was checked, appeared fine, and the paddle wheel was changed anyways.  The solenoid part of the valve was replaced (valve body not changed), which solved the problem.  Flow returned to 0.7gpm and the collimator thermocouples returned to normal value.  It is suspected that the valve was fluttering open and closed causing the decreased flow observed before, and then the valve finally failed closed at 1:30 on the 26th.

The following work was performed on the T2-MK2 target assembly during the 2017 fall shutdown:

Oct 4, 2017:
- Target moved from beamline to hot cell, measured 661mSv/hr at 0.5m (move went smoothly, having camera views at N/S and E/W angles very helpful)
- Flask tray wet wiped after transfer: 1st wipe 600cpm on 44-2 monitor, 2nd wipe 900cpm (3rd swipe not taken because tray looked clean)
- Flushed target with clean water, then compressed air (positions 0 through 5)
- Male Hansen fittings on the target assembly swapped with new units (in process of replacing all self sealing fittings in T1 and T2 water system)
- Checked all micro-switch positions and potentiometer readback: switches ok, pot value jumpy around position 1 and 2 (known from before), otherwise fine
- Left target overnight at position 4 with 20psi supply air running through to dry water system

Oct 5, 2017:
- Pumped down target with cold trap (~2 hours)
- Baseline leak check performed: 6 mins to reach 5.0E-3 Torr (3.2E-9 Torr*L/s), 21 mins to reach 1.0E-3 Torr (1.4E-10 Torr*L/s)
- Helium spray at 3psi for 0.5s bursts, tested with ladder in position 4
- No leak at any target positions or plugs
- Small leak on left side bellows, or bellows seal to ladder: up to 8.0E-9 Torr*L/s response, then quick recovery  (known from previous leak checks, same magnitude)
- Small leak on right side bellows, or bellows seal to ladder: up to 2.0E-8 Torr*L/s response, then quick recovery  (known from previous leak checks, same magnitude)
- Beam spot photos taken, position 1 (only target used this running period) has fairly well centered beam spot (see below)
- Position 4 target removed ("JAN29'97"), was very difficult to remove because one water tube stuck in ladder (this target was installed on ladder in 1997!  First beam 2014)
- Used hammer in hot cell to remove which eventually broke the target water tubes from the wire EDM housing
- sealing surfaces cleaned with burgandy Scotchbrite on electric dremel tool
- Inspected sealing surfaces and ladder fitting threads using Nikon level: all appeared clean and undamaged
- Installed new 12mm beryllium target at position 4 ("Jul.06#4"), no issues with install
- Leak check repeated: reached 1.0E-3 Torr after 3 minutes (1.3E-8Torr*L/s)
- Down to 3.5E-9 Torr*L/s after 18 mins, began test
- No response at all target positions, checked left side bellows: same response as above
- Beam spot profile at Position 1 measured with Nikon level
- Vertical and horizontal positions of new Position 4 target measured
- New profile monitor cassette retainer pin installed (plate hole had to be opened up to 0.375" diameter), used vice grip pliers clamped onto clevis pin in order to install

Oct 6, 2017:
- Profile monitor actuation tested: start movement 5psi, fully IN at 35psi
- Profile monitor limit switches tested, ok
- Target transported back to beamline, no issues
- BL1A pumped down, Edi reported pump-down normal
- Electrical check performed on T2 profile monitor from mezzannine: Sheng-Li reported results normal
- Cooling system started: no change to BL1A vacuum
- Target info sheet and target elevations sheet delivered to Main Ops
- XTPAGE 7L information updated

 The #6 male Hansen fittings for the T1-MK1 and T2-MK1 targets (both currently stored in the target storage pit) were replaced today with new units.  Some of the target assembly Hansen fittings had shown signs of leaking and the internal seals are not easily replaced.

"120V" warning labels were put on the micro-switch IN and OUT limit sensors for the profile monitors on both targets.  These are the only micro-switches on the assemblies that still run at 120V after the PLC controls upgrade.

test 2

 The three failed T2 cooling system secondary side solenoid valves were tested yesterday to try to determine the problem.  The test steps are detailed below:

• New spare valve (NSV) produced audible "click" and valve was seen to physically open when 24V DC driving voltage was applied
• Failed valves of July 25, 2017 (J25), July 10, 2017 (J10), and November 2016 (N16) all did not click or open when driven
• The solenoid of NSV was removed from its valve and installed on each of the failed valves in turn.  The NSV solenoid succesfully opened each of J25, J10, and N16 (the valve bodies and pistons of the failed valves appear to be fine, and the solenoids all faulty)
• Each solenoid's coil resistance was measured: NSV = 187kOhm, J25 = 200kOhm, J10 = O/C, N16 = O/C

On the next maintenance day Tony Tateyama is planning on performing continuity tests on the T2CS:SVSEC control cable and comparing it to the control cable for identical solenoids in the system.

 At approximately 13:50 today the T2 cooling system secondary side solenoid valve (T2CS:SVSEC) failed to a closed position.  This was the same location where the valve failed on July 10th (E-Log #224), and also in Nov 2016 (E-Log #211).  This repeated failure does not seem like a coincidence, so instead of replacing the valve with a new unit, it was bypassed temporarily until we can determine the cause of the failures.  A 3/4" FNPT coupling was installed in the place of the valve, and a CPC-4 stub was plugged into the control cable with pins 1 and 2 shorted in order to prevent the control system from timing out.  The cooling system has now been restarted and water is flowing properly through both sides of the system.  These failures must be investigated further during the fall shutdown, and ideally a solenoid valve should be reinstalled at this location (currently to shut off water supply to the secondary side the 1A tunnel must be entered to access the CuALCW supply valve).

Prior to shutting off the system for repairs some tests were performed by Tony Tateyama who provided the following summary:  "I wanted to see if there was any possibility of anomalous signals for T2CS:SVSEC drive & loop-back voltages at the PLC BOP.  Approximately 23.8 VDC was seen for the drive signal & slightly less for the loop-back signal.  In addition, about 70 mVAC ripple was also observed.  The same measurements were done for the T1CS:SVSEC voltages for comparison.  Approximately 23.9 VDC measured for the drive signal, and again, slightly less (~23.8 VDC) for the loop-back signal.  About 100 mVAC @ ~8 kHz was also seen. "

The T2 heat exchanger secondary side solenoid failed on July 9 at approximately 19:30 (EPICS ID: B1A:T2CS:SVSEC).  The valve was replaced with a spare and the cooling package is now running normally.

The valve at this location also failed in November 2016:  https://elog.triumf.ca/TIS/RH-Meson+Hall/211  Further investigation will be attempted to determine if there is a root cause to the failure, or if it is a coincidence.

The attached OneNote PDF contains details of the work.  Cyclotron Fault #10119 has been returned.

 On Thursday May 11th operators attempted to move the T2 profile monitor (T2-MK2) to the IN position but were unsuccessful.  The monitor would leave the OUT limit, but would not reach the IN limit.  This was repeated 2 times. The air amplifier was inspected and nothing out of the ordinary was noted.  The T1 profile monitor was tested and was working properly.  At the time it was suspected that the monitor cassette had become dislodged as occurred earlier this running period (see E-Log #221).

Today monitor actuation was attempted again and it moved to the IN position normally, this was done a total of 3 times..  An electronic check was performed by Mike Russell who reported that it looked the same as when he last checked, meaning the monitor cassette was definitely not unplugged. The monitor was then used with beam and Eric Chapman reported that monitor was functioning normally.  With the beam spot centered vertically on the profile monitor the protect monitor showed more beam on the lower plate, which is expected because the protect monitor center is 2-3mm higher than the profile monitor center (see E-Log #220).

It is unknown why it would not actuate properly last week, but seems to be fine now.  Cyclotron fault #9928 has been returned.

 During the target move on Thursday May 4th the paddle wheel and paddle wheel shaft for the T2CS:FGTGT sensor were changed to address transient spikes of approximately +/-1.0 gpm which were causing nuisance trips.  There have been no transient spikes recorded since the change, and the nominal flow value dropped from 4.9gpm to 3.6gpm.  Fault report #9836 has been returned, and the warning and trip levels, which were set higher to avoid nuisance alarms, were returned to nominal values.

We will continue to swap T1 and T2 flow meter paddle wheels every 2 years, which has been a suitable replacement interval besides this one case.

 On May 5th the T2-MK2 target was moved to the hot cell for the purpose of checking the beam spot on the Position 2 beryllium target to confirm the results of the T2 area realignment detailed in E-logs #218 and #220.  In preparation beam was put on this target overnight for approximately 1.5hrs at 10uA and 4 hours at 50uA (previously had not seen beam).  Also, there had been issues with the T2 profile monitor over the last couple days, so inspection and repeat of electrical check in the hot cell was planned.  The field in the T2 monolith work area was up to 1.3mSv/hr, and a maximum field of 230mSv/hr was measured 0.5m from the target during removal.

Unfortunately no beam spot was visible on either the entrance or exit windows at Position 2  (a longer exposure will be required next time a beam spot is to be inspected).  Detailed photos were taken, one of each side attached.  It was noticed that the T2 profile monitor cassette had become dislodged within its housing (it was sitting approximately 1" from the fully installed position - photo attached).  It is suspected that this happened when the target assembly was in the beamline - the first time the profile monitor was exercised back in April it moved quite quickly which may have caused the cassette to come loose (it is only held in with friction of the 9-pin connectors and the fit of the cassette body in the housing).  It was re-inserted in the hot cell and an electrical check was performed by Shengli: all good.  After the target was returned to the beam line the electrical check was repeated while the monitor was actuated in and out: once again all good, and the movement of the monitor was slow and smooth (video taken)

Prior to this job new cameras were installed in the hot cell opening aimed at the support ring hole (one aimed north, the other east).  This proved to be a very useful upgrade and made the installation process significantly easier and smoother.  Two cameras were also used during target installation at the monolith (only 1 was used in the past).  Having a second camera at a different angle made alignment easier (reducing time and dose), and should be adopted for future target moves.

Vacuum pumpdown was normal, the cooling system was restarted without change to the vacuum, and operators were able to use the T2 profile monitor during beam ramp-up overnight.

Dose for this job:  Isaac 0.17mSv, Keith 0.06, Ron 0.06

The following work was performed on the T2-MK2 target assembly and the T2 Protect assembly for maintenance purposes and to implement the alignment strategy detailed in E-Log #218:

March 30 (T2-MK2 target in hot cell):
- Profile monitor actuation checked:  moved in and out smoothly, fully in at ~25psi
- Ran through all target positions, motion smooth, micro-switches all ok, potentiometer value unsteady when moving between position 0-1, steady when stopped, no issues form positions 2-5 (this was also observed last shutdown and there were no issues with ladder moves in the 2016 running period using EPICS)
- Water system pumped out with cold trap for 2 hours (previously was purged with clean water, then air overnight)
- Leak check performed: 10 minutes to reach 5.0E-3 Torr TP pressure, 25 minutes to reach 1.0E-3 Torr; 3psi helium supply applied in 0.5s bursts;  No response at all target locations and all four aluminum delta seal locations.  At base of bellows on left side (when facing ladder) quick response up to 0.6E-8atm.cc/s, quick recovery to baseline (<10s); At base of bellows on right side (when facing ladder) quick response up to 1.2E-8atm.cc/s, quick recovery to baseline (<10s).  These small helium leaks were known (see E-Log #95) and have not caused issue in multiple years of running.
 

April 5:
- Position 1 and 2 potentiometer ratio changed to achieve new target alignment, look-up tables updated in PLC code (see E-Log #218, and attached new elevation measurements)
- Position 1 and 2 micro-switches adjusted to be centered at new positions
- Profile monitor inspected: no broken wires, but some not quite straight or parallel with others, also some wire insulation blackened.  Scott Kellogg recommended installing new profile cassette
- Old profile cassette removed in hot cell and new one installed
- Profile monitor elevation adjusted (see attached new elevation measurements)
 

April 6:
- Electrical check on profile monitor performed at HC by Dave Cameron: deemed OK
- T2-MK2 moved from the hot cell to the beam line (~200mSv/hr at 0.5m)
- T2 protect monitor moved from beam line to hot cell (~16mSv/hr at 0.5m)
 

April 7:
- Elevation of protect monitor measured to confirm old measurement: within 0.1mm
- Old protect monitor cassette removed
- New cassette supplied pre-adjusted by Scott Kellogg
- New cassette installed in HC, elevation measured on entrance side: 1832.8mm (intended elevation was 1833.2mm, from E-Log #218)
- Protect monitor electronics checked by Shengli Liu and Dave Cameron at the hot cell: all OK
 

April 10:
- T2 profile monitor cable plugged in which caused unexpected jump in vacuum: 20mTorr - 180mTorr
- Large nut at top of feed-through connector tightened and vacuum returned to baseline
- T2 target position 1 and 2 potentiometer values tested from EPICS system to confirm PLC look-up table changes ok
 

April 11:
- T2 Protect assembly returned to beamline (12mSv/hr at 0.5m).  T2 monolith position was monitored by Beamlines Group with their laser tracker during the move.  A maximum of 0.2mm movement was measured which is attributable to noise.
- Air amplifier started, leak found from one of the M20BB cylinders;  went away after reset of BB control system, BB exercised several times and leak could not be reproduced (no change to BL vacuum)
- T2 profile monitor tuned for slower actuation, smooth movement confirmed and no change to beamline vacuum (also done for T1 profile monitor)
- Electrical check on T2 protect monitor performed by Dave Cameron and Shengli Liu from racks on mezannine: all OK
- Electrical check on T2 profile monitor performed by Dave Cameron and Shengli Liu from racks on mezannine: initially very low signal, they discovered a previously unknown large resistor which was bypassed for the test:  approximately 1/2 of signals show response to HV blip, deemed OK to restore shielding
- Target info sheet and updated elevations measurements delivered to Operations (elevations sheet attached)

Note: the T1 target (T1-MK2) was not removed from the beamline this shutdown, so new elevation measurements have not been submitted and T1 profile and protect monitor checks have not been performed

 The following work has been completed on the T2 cooling package:

- T2 system drained to South TNF holding tank (completed in Jan 2017, water already released)
- Target water and demin. circuit Cuno filters changed
- Both female Hansen fittings at target end of flexible hose replaced with new SS models (zinc coated steel models (PO# TR207041) were accidentally ordered and installed last shutdown and had black corrosion inside)
- Small leak on pipe thread fitting near target water supply solenoid repaired
- Reservoir o-rings replaced
- System filled and re-started, no leaks observed, no drop in water level over 48 hour run period
- Demin. circuit flow rate tuned to 1.0gpm

A strategy for alignment of the T2 protect monitor, T2 target, and T2 Collimator A was discussed today and decided upon by Yuri Bylinski, Isaac Earle, Yi-Nong Rao, and Roman Ruegg.  The purpose of the adjustment is to try to reduce the amount of beam lost as it passes through the T2 area (presumably being blocked by the collimator because it is not possible to steer the beam through all three devices in their current configuration).  Keeping in mind that changing the elevation of Collimator A would be prohibitively time consuming it was decided that the protect monitor and target both be adjusted upwards 3.0mm compared to where they were during the 2016 running period.  This will result in the following elevations (all w.r.t. the T2 target flange).  The T2 profile monitor elevation will also be adjusted to match the target elevation.  

T2 Protect Monitor:  1833.5mm (1833.2mm from protect monitor flange)

T2 Target: 1836.5mm 

T2 Collimator A:  1834.7mm  (1834.5mm from collimator flange;  This is the current collimator elevation, measurement taken by Tom Lyth in Feb 2009)

These values are similar to what were used in 2009 during which time TNF neutron flux rates were acceptable (>800), and centered beam spots were achieved.  During the 2016 running period the beam spot on target was also roughly centered.  At both times the target was approximately 3mm lower than the protect which seems to indicate the beam is travelling at a downwards angle, possibly not correctable with available tuning.  This is a departure from the strategy in the last few years when the target and protect monitor were adjusted downwards to bring them closer to the beam plane (they had risen due to movement of the T2 monolith).  This is now thought to have been a mistake because it made it difficult to steer beam through the collimator.

Beam will be started with T2 in Position 2 (new target, never seen beam), then after a few days the target assembly will be moved to the hot cell to check the beam spot to confirm alignment and make changes if the strategy needs adjusting.

A PDF copy of Yi-Nong's summary is also attached.

 The following changes were made to the T1/T2 PLC control system:

- De-glitching system investigated and Tony discovered that it was programmed incorrectly:  the de-glitch time was observed only when a PV changed from bad state to good, not the opposite (meaning there was actually no de-glitching in place since system start-up in 2016!).  This was reversed and the following de-glitch times were programmed:  100ms for trip thresholds except 300ms for flow meter trips, and 2s de-glitch on all warning thresholds.  The new system was tested with the PLC in simulation mode and seems to be working correctly.  This should eliminate or significantly reduce nuisance warnings and trips from the system.

- PLC system code was modified so that the target version displays '???' when no target is installed instead of defaulting to 'MK1'  (all target positions registered as 'plugged' with no target installed, so there was no danger to the system, but this was still confusing to operators)

The 18' shielding block which lies west of the T1 cooling package was removed on Friday Mar 24 so that vacuum equipment could be craned into the 1A tunnel.  During replacement, the flexible SS hose for T1 target return water got caught under the block and was ripped apart.  Workers in the area did not realize this had occurred.  The cooling package tripped off when the hose ripped (likely due to low water flow or low supply pressure).  In total approximately 6.5L was lost which flowed to the 1A tunnel below.  The hose connections at the target and cooling package do not appear to have sustained damage.  Plans are under way for a replacement hose to be fabricated and installed.

Update March 30, 2017:  The damaged hose has been replaced with a new 20' long braided SS hose with 3/4" male NPT fittings both ends.  Both air supply lines for T1 profile monitor actuation (1/4" copper) were also damaged during the Mar 24 block move and they have now been replaced.  All hoses, cables, etc on the west side of the T1 monolith have been re-routed or secured to reduce the chance of this happening again in the future.

 On March 13th in the morning the Meson Hall Hot Cell Lab active sump reached the high level sensor.  The system responded as expected:  high level panel warning light illuminated, siren sounded, and MCR notified via "B1 RCR1 RM6 WARN".  A 1L sample was delivered to RPG for analysis.  After approval, the sump was pumped out over the following two days.  The procedure in Document #64834 was followed for sampling and pump-out.

The following work was completed on the T1 cooling package:
- water drained to BL1A holding tank
- all resevoir o-rings changed
- all cuno filters changed:
     M15 permanent magnet filter < 5uSv/hr at 0.5m, dark grey colour
     Target inlet filter 210uSv/hr at 0.5m, light brown colour
     Resin filter, <5uSv/hr, looks clean and new
- package re-filled and started
- demin circuit flow tuned to 1.0gpm
- #6 female Hansen fittings at target end of flexible hose swapped for previously used SS models (zinc coated steel models (PO# TR207041) were accidentally ordered and installed last shutdown and had black corrosion on the inside)
 

The T2 cooling package was also drained to the 1A holding tank.
After approval from RPG the holding tank water was released.

Note: the appearance of the used cuno filters is consistent with previous years.

 The beam spot on the 10cm beryllium target in Position 1 on T2-MK2 (Target ID: "OCT.06#3") was measured and photographed yesterday.  A fairly well centered spot was measured to be approximately 4.6mm wide x 5.2mm tall on the entrance window and 5.7mm wide x 7.6mm tall on the exit window using the Nikon micrometer level.  This target first saw beam on June 16, 2016.

A PDF scan of the beam spot measurements and photographs of the beam spots are attached.

 The T2-MK2 target assembly was moved from the beam line to the hot cell today.  A maximum field of 203mSv/hr was measured at approximately 0.5m

A steel shielding plug has been installed in the T2 hole to reduce shine and allow the T2 volume to be pumped down.

 At approximately 11am today the T2 cooling package heat exchanger secondary side water flow suddenly dropped from ~11gpm to 0.  This was confirmed to be a real reading by observing that with no secondary cooling both the inlet and outlet target water temperatures rose steadily with beam on the target.  While standing on the blocks in the T2 area and having a controls expert (wither "superuser" access mode) actuate the secondary side solenoid independently it was confirmed that no "click" was heard when power was supplied to the valve.  No other changes were observed with other parts of the CuALCW system, therefore it was concluded that the secondary side solenoid valve (B1A:T2CS:SVSEC) had failed.

Blocks were removed above the T2 cooling package, the CuALCW lines were valved off and drained, and the failed valve was removed.  A new valve was installed.  A section of 3/4" SS tubing was replaced with flexible 21" long 3/4" diameter SS hose to allow re-installation (alignment could not be achieved with rigid tubing).  Water supply and return were restored, and the system was started.  Because of the new section of tubing, the flow through the secondary side dropped from ~11gpm to ~9gpm.  This should not have a significant effect on operation.  No leaks were observed.

Total dose for Isaac: 0.15mSv, for Keith: 0.03mSv

 An expansion tank high level warning has been added to the T1/T2 control system under Work Request #4011.  The purpose for this new warning is to alert staff of an unexpected water increase in the unlikely scenario that a leak develops in the heat exchanger between the two fluids and CuALCW on the secondary side enters the target water system.  The new thresholds for high level warn, low level warn, and low level trip have been set at 40cm, 30cm, and 25cm respectively.  The nominal water level is 35cm.  All thresholds were tested by adding water to the tank, or removing it.  All responded as expected.

 The T1 cooling system inlet pressure transducer (EPICS ID: B1A:T1CS:PGIN,  Omega part # PX315-100GI) began failing in August (exhibited gradual decline in pressure with no corresponding change to flow rate or outlet pressure).  A new transducer was installed on Sept 30 2016 which fixed the problem.  Cyclotron Fault Report #9238 was returned and the temporary bypass on the warning level was removed.

 The following work was performed on the T1-MK2 target assembly during the September 2016 Mini-shutdown:

Monday Sept 26:
- Target moved from beamline to hot cell (28mSv/hr at 0.5m)
- Position 4 beam spot on entrance and exit windows photographed (attached)
- Target flushed with fresh water, positions 0, 4, and 5 blown out with air, then left overnight with air blowing through position 0 at 10psi supply
 

Tuesday Sept 27:
- Pumped on target water system using lab leak detector:  reached 1.0E-3 Torr and E-8 Torr*L/s range in < 5 mins
- Initial leak check performed: TPP < 1.0E-3 Torr, Base leak rate: Low E-9 range, 3psi helium supply, ~0.5s bursts
- Response up to 3.6E-6 at lower left side delta seal (when facing ladder), up to 4.9E-7 for lower right seal  (note: T1 area at good vacuum level prior to mini-shutdown),  All other positions no response, or response in low E-8 range (assumed drift or insignificant helium leak)
- Plugs at positions 1, 2, and 3 removed.  All threads inspected, brushed with wire brush, rinsed with acetone, and blown dry with compressed air
- New targets installed:  Serial #101 at Position 1, #102 at Position 2, and #103 at Position 3  (Position 1 threads significantly stiffer than others, but no major issue)
- Leak check repeated: similar response from all locations including all beryllium target windows  (concluded suitable for use in beamline)
 

Wednesday Sept 28:
- Profile monitor actuation checked in hot cell:  movement starts at 27psi, fully actuated at 48psi, limit switches functioned properly
- All target ladder micro-switches checked, ok
- Vertical and horizontal target position measurements taken using procedure in Document #50194  (PDF attached)
 

Thursday Sept 29:
- Target returned to beam line and all services connected
- Target info sheet and elevations delivered to Ops, XTPAGE 7L updated
- T1/T2 PLC information updated with new cassette information
 

Friday Sept 30:
- Vacuum pumped on the T1 volume.  Quickly 200mTorr (threshold for T1 Vac OK signal).  Edi reports pump-down slower than usual, will evaluate on Monday.
- T1 package started (no change in BL1A vacuum)
- Profile monitor actuation checked from control room (actuated normally)
- T1 profile monitor and protect monitors checked by Mike Russel and Victor Verzilov:  profile monitor normal, protect monitor signals low but Victor expects it is fine and give ok to cover

 

To address the deficiencies identified in E-Logs #204 and #205 the old active sump high level switch was replaced with Omega LV164SS model (316SS stem and float, 4' length, standard mount) using adaptor flange TRH1316 to mount to the top flange of the sump.  A new electrical box was also installed (circuit diagram attached in PDF format).  The circuit was designed by Don Jackson and wired by Travis Cave.  Connection to the RCR Lab PLC was done by Dave Morris.

The switch was tested using the following procedure:

1.  The switch was triggered by submerging the base in water until the float lifted.  This was confirmed to be equivalent to disconnecting the CPC-4 connector for the switch from the new electrical box
2.  Switch connector disconnected:  Local alarm sounds in HC Lab, red "Sump High" LED light illuminates, green "Sump Good" LED turns off, RCR PLC reports high level (cannot be reset), main control room "MHESA B1 RCR1 RM 6 WARN" displays NOT OK  (this will sound every 3 minutes until the RCR PLC high level alarm is reset)
3. "Alarm Silence" button pressed on new electrical box:  Local alarm stops, LEDs on electrical box do not change, RCR PLC still cannot be reset
4. Switch connector reconnected:  "Sump High" LED turns off, "Sump Good" LED turns on, RCR PLC can be reset
5. Power to the electrical box removed by switching off the breaker:  All LEDs turn off, no alarm, RCR PLC reports high level

Pump out of water from the MHESA Lab active sump to the HC Lab active sump was tested while observing the HC Lab sump.  The water entering the HC lab sump did not splash on the new switch excessively, and did not cause any false alarms.

The new system will continue to be tested annually according to the calibration schedule.

 A new version of the T1/T2 PLC code has been uploaded to the PLC CPU.  The main change in the new version is a fix to a bug related to the target MK# which may have caused the previous CPU crashes.  The EPICS interface now needs to be updated so that no target information is displayed on the EPICS screens when there is no target installed, or jumpers for both MK1 and MK2 are detected.  This must be tested during the next maintenance day.  The appropriate tests are Section 6.4.1 Test #5 and #6 from Document #118467

Further tests were done on the active sump high level float switch today.  The switch was triggered 1 time by filling the sump, and approximately 8 times by tilting the sensor manually.  A local alarm sounded in the hot cell lab each time, but only one time in the RCR Lab and MCR (this occurred on the final trial).  Fault report #9099 has been submitted to Controls Group.  Although the existing float sensor did cause a local alarm when the sump was filled, it's operation did not seem ideal:  it only triggered when completely horizontal, which wasn't until water was several inches above the base of the switch;  in addition it appeared that it may be possible for the switch to become stuck under the horizontal portion of the sump (possibly even in an untriggered orientation).  For these reasons a replacement level switch will also be investigated.

UPDATE (Sept 8, 2016):  Further testing revealed that the electronics for the high level switch were wired incorrectly which resulted in the RCR PLC only seeing sump level high when the "silence" button on the electronics box for the switch was pushed.  This was not discovered during previous tests because the "silence" button was pushed each time shortly after triggering the switch.  The new electronics for the switch will address this issue.

 The high level sensor for the Meson Hall Hot Cell Lab Active Sump was tested today and worked as expected.  When the sensor is tilted (as occurs when the water level rises above it) an audible alarm sounds in the hot cell lab, and the Safety Panel in the MCR also alarms with "MHESA B1 RCR1 RM 6 WARN" displaying NOT OK.  This alarm sounds every 3 minutes in the MCR until the PLC outside of the RCR lab is reset.  When the sensor was tested on the RCR lab PLC there was a message indicating that the MH Hot Cell Lab Active Sump is full.

It is still unclear why the operators did not receive this alarm when the sump filled and overflowed last night (or possibly they did receive it, but it was ignored or silenced).  The operators from that shift are currently off until this Saturday, so they will be contacted next week to try to learn more.

UPDATE (Sept 8, 2016):  Further testing revealed that the electronics for the high level switch were wired incorrectly which resulted in the RCR PLC only seeing sump level high when the "silence" button on the electronics box for the switch was pushed.  This was not discovered during previous tests because the "silence" button was pushed each time shortly after triggering the switch.  The new electronics for the switch will address this issue.

 A call was received from the MCR at approximately 21:30 on Monday June 27th to report that active drains in the east end of the Meson Hall were backing up and overflowing.  I came in and found the leak coming from a broken spray nozzle in a sink in the rad waste area.  The leak was stopped at which point there were puddles between 1-2m diameter around the active sump top flange, the drain directly south of the warm cell water windows, and the drain in the counting room south of the clean room.  There was lots of water on the floor in the rad waste area as well.  No water was found in the TR-13 area.  A swipe was taken from the puddle around the active sump top flange, which was not above background level.  Robin Swanson (ops) contacted Joe Mildenberger who gave permission for the sump to be pumped out after a water sample was taken.  Robin took a water sample and left it for analysis by RPG, the sump pump was started and ran overnight.  At approximately 8am the next morning the sump was almost empty and the pump was stopped.  Floor swipes were taken where there had been puddles which were now dry - no contamination found.  The areas in the hot cell lab which had been wet were mopped.

 With-beam commissioning tests were completed today by Tony Tateyama and Isaac Earle.  The test results are attached in PDF format.  These tests are described in Section 7 of the Commissioning Plan (Document-118467 Release #2).  These tests were not part of the official commissioning, but intended as an additional check of system functionality with beam on, and to confirm that the CCS current protection system performs as expected.

Official commissioning of the T1/T2 EPICS system was completed in April 2016, with the results released in Document-131503.

 Position #4 on the T2-MK2 target has now reached the maximum beam time.  The target was moved to Position #1 at the nominal elevation.  This target should be used for the remainder of the 2016 running period.

The T2-MK2 target at Position 4 has been adjust to approximately +1.5mm offset.  This causes the ladder to be moved downwards, raising the beam spot location on the target window.  The potentiometer ratio at this offset is approximately 0.810 (> 0.8059 nominal ratio for position 4, and therefore closer to Position 5)

The purpose of the offset is to better center the beamspot on the target window.  Target window photographs showed the beam spot being slightly below center (See E-Log #197), and target ladder elevations taken in Spring 2016 showed the Position 4 target approximately 1.0mm above the profile monitor center.

MCR has been instructed to continue running with this offset, and a message with details on how to do so was posted to their Plone site.

The Remote Handling Hot Cell Lab Active Sump has been released to city sewer following the procedure in Document-64834.  The water was close to the sump high limit switch (approximately 5800L).  Before release the water was sampled, tested, and approved for release by RPG.  The tritium analysis form has been filled out and returned to RPG,.

It took approximately 5 hours to empty the sump with all valves fully open and the water flowing through the 10" Cuno filter.

Mike Russell from Diagnostics Group has completed an electrical check on the T1 and T2 profile and protect monitors.  He reports that both profile monitors look fine.  The T1 protect monitor looks fine except for a weak or missing signal from the halo monitor.  The T2 protect monitor looks fine.

Note: The protect monitor on the T1-MK2 target was working without issue during the previous running period and was not removed from the beam line during the 2016 shutdown.  The cable for the T1 protect monitor was not disconnected during the 2016 shutdown.

On April 11th the T2-MK2 target was flushed with water, then overnight with air at 5psi supply pressure.  The target ladder was in Position 0 for this procedure.

On April 12th the target was pumped using a cold trap and the Varian leak detector for approximately 2 hours.

On April 13th pumping was restarted and the leak detector reached 0.0E-10 Torr*L/s with < 1.0E-3 Torr test port pressure.  A general helium spray was performed with a small response to the mid 10^-9 range and quick recovery (this target was in use all of the 2015 running period w/o any vacuum issues).  Plugs were removed from Positions 1 & 2.  The swagelok threads on the ladder were cleaned with a wire brush then pressurized air.  The sealing surfaces were cleaned using Scotchbrite then blown with air.  Threads and sealing surfaces were inspected with the Nikon level: some minor thread imperfects on the threads and a few small pits on the sealing surfaces were observed.  New 10cm beryllium targets were installed (Position 1:  Target "OCT.06#3",  Position 2: Target "#201").  A leak check was performed using 0.5s bursts of helium at supply pressure 3psi.  Helium was applied at all target positions, delta seal locations, and along the length of both bellows.  No responses observed that were greater than the initial leak check performed before the plugs were removed, therefore the target was deemed water leak tight and ready for use.

On April 14th position measurements for both new targets were taken as per Document-50194, which will be included in the target info sheet to be delivered to Operations.

On April 15th the target was returned to the beamline.  All services were attached, the cooling package started, and vacuum pumped down normally.

Photographs of the Position 4 beam spots as well as the target ladder after new target installation are attached.

The T2-MK2 target was transported from the beam line to the hot cell using the transfer flask.  The target was measured to be 120mSv/hr at 0.5m.  The steel shielding plug was installed in the T2 target hole to reduce upwards shine.

New beryllium target cassettes will be installed on the target ladder before it is returned to the beam line.

 The lower limit of the flask hoist hook has been adjusted.  The travelling nut for the lower limit was rotated 3 notches closer to the upper limit travelling nut which resulted in an additional 1-3/8" travel of the hook at the lower limit.  The door hinges and door cable pulleys were lubricated with WD-40 and actuation of the doors was tested several times.

The T1/T2 Control System "Assembly Tests"  (Document 118467 Release #1, Section 5) have been completed.  The expected result, as written in the document, occurred for every test.  The T2 Shield Thermocouple #3  (B1A:T2:TC3SHLD:RDTEMP) showed slightly erratic behaviour when viewing archived data.  It is uncertain at this point whether or not it will need to be bypassed in the B1A:T2:TGTRDY interlock.

The official commissioning tests (Section 6) are currently on-going.

Manufacturing of a new batch of T1 and T2 beryllium targets has been completed.

The job was completed under Machine Shop Work Order #41527.  Eleven 12mm T1 targets (TBP1683) and seven 10cm T2 targets (TBP1691) were completed (exhausting all stock of the leftover EDM'd target housings).  The target window profiles (TBP1695) were laser cut at an outside shop for greater precision and to save time.  The target window assemblies were electron beam welded to the target housings (TBP1687 & TBP1692) by Bob Welbourn.  The remainder of the tube welds were completed by George Sun.  Assembly of the Swagelok fittings, target inspections, hydrostatic testing, and final helium leak checks as specified on the assembly drawings were performed by Isaac Earle.  In addition to these tests, one spare target housing was pressure tested to 275psi (the max pressure of the regulator of target jig TBP1721).  The target windows bulged at this pressure, but did not rupture.

The attached spreadsheet has details on target measurements and test results.  The target serial numbers (listed in the spreadsheet) have been engraved on the 90 degree elbow of each target on both the entrance and exit side.  Detailed photographs were taken of each target which are stored on Isaac Earle's hard drive (too large to attach).

If the 2015 target replacement plan is followed (See E-Log #168), these targets should last for at least 10 years.

Fabrication of the T1/T2 beryllium target testing jig (TBP1721) was completed today by Dan MacDonald.  With a target installed across the test ports, the inside volume pressurized with air to 103psi, and both ball valves closed, there was no noticeable drop in pressure over a 16 hour period.  This result was deemed satisfactory for use of the jig for hydrostatic testing of new targets.

Three used TNF / 500MeV system cooling system resin cans were removed from the Meson Hall target storage pit (two from Hole #1, one which was stored below block #1).

Two new active cans will be returned to Hole #1 after the TNF resin is changed in a few weeks time.

Preliminary testing of the new PLC control system and EPICS interface was completed today for the T2 target, T2 cooling package, T2 collimators, and T1 thermocouples.  All systems behaved as specified.

Review of the commissioning plan is currently underway.

A scanned copy of the test plan is attached.

 Preliminary testing of the new PLC control system and EPICS interface was completed today for the T1 target, T1 cooling package, and T1 collimator.  All systems behaved as specified.  This test did not include testing of any T1 thermocouples which will be included as part of the T2 test program to follow.  This is not considered system commissioning, which will follow after all preliminary testing is complete and the commissioning plan document has been released.

A scanned copy of the T1 test plan with notes is attached.

 A 1/2 turn twist in the target flask hoist chain has been corrected to address a deficiency observed by inspectors by Gordon Crane.  This was accomplished by first holding the flask door yoke at the top position using rope slings, then lowering the hook block spider approximately 6", then lifting the stationary end of the hoist chain approximately 2" using the Meson Hall main crane to make the top of the chain slack.

A missing cotter pin for the counterweight on the RH HC Lab overhead crane was also installed.

The following work was performed at the T2 cooling package and target between Jan 25 - Feb 3, 2016:

- Cuno water filters removed (2), and new filters installed
- Target water solenoid, H.E. secondary side solenoid, Collimator A solenoid, and Collimator B solenoids removed and replaced with new 24V DC versions
- All thermistors removed and replaced with Type K thermocouples
- All Proteus paddle wheels, o-rings, and paddle wheel pins replaced (5 sensors total)
- Target side #6 Hansen QD fittings replaced
- All air QD fittings (6 total) for profile monitor actuation were replaced, some of the flexible hosing was replaced (cracking), and the flexible hose assemblies were checked for gross air leaks
- Top surface of the cooling package resin can support platform was painted (was rusty)
- Resin can swapped (previous resin swap was during Shutdown 2014)
- Target water check valve was removed and tested (working as expected), then reinstalled

The following work was performed at the T1 cooling package and target between January 18-21, 2016:

- Cuno water filters removed (3), and new filters installed
- Target water solenoid, H.E. secondary side solenoid, and Collimator A solenoids removed and replaced with new 24V DC versions
- All thermistors removed and replaced with Type K thermocouples
- All Proteus paddle wheels, o-rings, and paddle wheel pins replaced (5 each)
- Target side #6 Hansen QD fittings replaced because internal seal had failed and fittings were leaking when disconnected (2)
- Profile monitor copper air supply line leak was repaired
- All air QD fittings (6 total) for profile monitor actuation were replaced, the air restricting valves were tuned, and the flexible hose assemblies were checked for gross air leaks
- "T8" temp sensor was found to be located on the CuALCW common return line, not on the Col A supply line as labeled on XTPAGE.  Position was swapped

Total dose for this work was 0.17mSv for Isaac Earle, and 0.10mSv for Keith Ng

 The T1 and T2 cooling packages were drained to the BL1A holding tank on Jan 8 2016.  The volumes were 100L and 80L respectively.  Before starting there was 350L of water in the holding tank from the 500MeV system which Steve Sapriken drained in September 2015.  The T1 and T2 systems were sampled and delivered to RPG.  After receiving permission from RPG to release all 3 portions of water it was pumped into the city sewer on Jan 18, 2016.

Isaac Earle has recently been managing the fabrication of some new Be target cassettes for the T1 / T2 Meson Hall target ladders (WO 41527).

The machine shop has outsourced the cassette tubes to an external shop for wire EDM cutting, as it produces a much more repeatable and consistent wall thickness without the need for delicate forming and welding.

However, the legacy end-cap design TBP1690 calls out a thin-wall formed frame to help brace the window in place and provide material to fusion weld the rim of the end cap to the cassette tube.  The legacy design frame is extremely thin compared to the tube wall thickness, and this mismatch is extremely difficult to EB weld reliably.  If a mistake is made, the entire EDM cassette tube is scrapped.

Bob Welbourn and the machine shop have come up with an improved design for the frame and window with slightly modified forming tools.

The window thickness .003" and tube wall thickness .010" remain the same.  The only change is to the thickness of the welding frame, which is now matched with the tube wall thickness (previously .003", now .010").

Photos of the old (right) and new (left) weld results are attached, as well as the new forming tools created by Bob Welbourn.

The shop is now able to make a confident and reliable weld for the end caps.

I am not sure that the increased frame thickness will have any affect on the beam entry into the target cassette, but my assumption is that this region is not critical.  Now, if the beam does enter that material at the edge of the cap / tube, and the temperature changes during operation, there will be a more secure weld there due to this design improvement.

The machine shop was waiting for the OK to finish welding the end caps into the frames using this new technique / design, and seeing as how the old option was likely to produce more leaking or scrap parts, I gave them the go-ahead to proceed.

Isaac will take care of the drawing and documentation updates when he returns from holiday.

Thanks and regards,

Grant

The following work was performed on the T2-MK1 target.  The profile monitor and monitor cage on this target were damaged in January 2013, See E-Log #68

Oct 23:
- T2-MK1 target moved from storage pit hole #4 to the hot cell (field measurement was not possible because the pole monitor had issues
- Position 2 and 3 targets photographed (see attached)
- Position 3 target (10cm Be) removed.  This target had accumulated 11.5 months of beam time.  Maximum for T2 targets is 12 months according to new schedule.

Oct 26:
- Position 3 plugged
- Profile monitor cage inspected and photographed. The cage (TBP0106D) and aluminum frame (TBP0101C) were both noticeably damaged (see photos)
- Attempted to insert dummy cassette, went in smoothly
- Moved target ladder through all positions, moves smoothly, micro-switches ok, potentiometer ok
- Motor drive coupling inspected - no issues
- Actuated monitor in and out: traveled fairly smoothly, 40psi required to insert, 20psi to pull back out
- Dummy cassette removed, real replacement cassette inserted (went smoothly)
- Horizontal wires inspected w/ Nikon level, found to be crooked

Oct 27:
- Cassette frame nuts (TBP0108C-2) loosened and monitor frame adjusted until horizontal wires horizontal, nuts re-tightened and torqued
- Vertical and horizontal measurements of profile monitor position measured - All within 1-2mm of previous measurements in 2010
- Although some physical damage to frame is visible it still positions monitor in correct location and seems to be fully functional, therefore will use if electrical is ok

Oct 28:
- Electrical check performed on profile monitor by Mike Russell
- All 26 wires visible.  Screenshot of scope taken (see attached)
- Moved to position 5 to stretch bellows, and air left running through target overnight to dry

Oct 29:
- Cold trap filled with LN2, pumping with new leak detector started
- Test port at E-1 Torr range after 30s, E-2 after 1.5mins, E-3 range after 3mins, 0.0E-10 torrL/s leak rate after ~10mins, TP at <1.0E-3Torr after ~15mins
- Leak check performed with 3psi helium applied w/ ~0.5s bursts at all target locations, bellows, top of vac flange, and Hansen fittings
- Small leak found at left bellows lower weld (expect is negligible for water use), no response from all target locations (see attached scan of leak check notes), deemed water leak tight

Nov 3:
- Plumb bob measurements taken
- Vertical and horizontal measurements of Pos'n 2 target taken
- Target ladder moved to position 0, profile monitor secured in IN position to prevent damage during move

Nov 5:
- Target returned to storage pit hole #4
- Field of 53mSv/hr measured at ~0.3m (attempted 0.5m measurement, but was a little close)

The T2-MK1 target is now fully functional and ready for use.  The 10cm Be target in position 2 has not seen beam.

Note that the target window welds show some uneveness (see photos), but not expected to affect performance b/c leak tight.  The elevations sheet will be updated and filed in the T2 Binder in the Meson Hall HC Lab Office.

 The T1 and T2 targets were both moved from Position 0 to Position 4 at approximately 2:30am on Saturday October 10th using the portable control box.  Operations had tried to move the T1 target to position 4 using the old control system but for unknown reasons it would move to any position except position 4 and the portable control box had to be used.  The portable control box should continue to be used for T1 and T2 target moves until the new control system is installed in the 2016 shutdown.

When the damage to the T2 Protect Monitor was found to be caused by flying debris from a vacuum spike (see e-log #179), it was decided that the T1 Protect Monitor should also be replaced.

Oct 8:

- The T1-MK2 target was transported from the beam line to the hot cell after the T2 target was returned to the beamline (see e-log #180).  The T1 target was 38.9mSv/hr at 0.5m

- The T1 target was inspected in the hot cell: profile monitor looked ok, protect monitor damaged with holes and dents on upstream side (see attached photos), position 4 target looked ok with beam spots roughly centered (see photos)

- The old protect monitor was removed in the hot cell and a new monitor supplied by Probes Group was installed (previously undocumented brass spacers were seen over the monitor cassette alignment pins on the target)

- The T1-MK2 profile and protect monitors were checked by Probes Group at the hot cell --> both ok

- Measured elevation of the new T1-MK2 protect monitor: 1831.1mm on entrance side (previously 1831.0 --> ok)

- Position 4 target entrance side elevation checked: 1832.9mm (previously 1831.9mm --> 1mm discrepancy)

- Position 4 exit side elevation measured: 1832.4mm (previously 1832.1mm --> ok)

- Protect monitor exit side elevation measured: 1833.7mm (previously 1833.3 --> ok)

- T1-MK2 target returned to beamline at approximately 5pm, no problems

- Vacuum pumped down quickly and it was decided that a leak check would not be necessary

Oct 9:

- All target services attached

- T1 target moved to Position 0 using remote control box

- T1 package started, no change to T1 vacuum

- T1 profile monitor and protect monitor electronics checked by Probes Group from mezzannine --> both ok

- T1 motor drive cable conected to controller in 1A racks.  The T1 ladder can be moved from the 1A racks mimic panel (T2 still requires portable control box)

- Actuation of the T1 profile monitor was tested --> actuated properly, no change in vacuum

- T1 cooling package expansion tank level checked: no change

- The T1 area is ready to be covered

While the T2 monolith was uncovered for T2 Protect repair (see previous e-log), it was decided that the T2 target should be moved to the hot cell to address issues discovered last week with target ladder movement (see e-log #175)

Oct 7:

- T2-MK2 target was moved from the beamline to the hot cell after the T2 protect was reinstalled (the target measured 896mSv/hr at 0.5m)

- The target was inspected in the hot cell: profile monitor looked normal, two beam spots (one higher, one lower) were observed on the entrance window (see photos), the spot on the exit window was low but not quite touching the weld

- T2 profile monitor was checked by probes group at the hot cell --> ok

- Ladder movement was checked by cycling through all target positions twice using the portable control box.  This revealed that the drive motor is actually functioning properly and the fault lies elsewhere

- Tested T2 motor drive cable from 1A racks to beamline with Dave Cameron.  Found Pin B has 190kOhm short to building ground.  Attempts to fix the cable at both ends were unsuccessful and spare wires in the bundle could not be used because there is a break in the cable at an unknown location and use of wire colours in both sections is inconsistent

- Checked entrance side elevation: 1839.0mm with potentiometer at 16.19kOhms.  This is consistent with previous measurement of 1838.7mm.  This confirms that pot, motor, and drivetrain for ladder movement are all ok.

Oct 8:

- Decided to run new cable for target motor drive  (already had on hand for T1/T2 controls upgrade in 2016 shutdown).  The cable is labeled T2TGTMTR-DRV.  It was first tested at the hot cell and then installed by Doug and Ron

- The T2-MK2 target was returned to the beamline.  A camera and monitor setup was used to watch the target entering the hole remotely which greatly reduced dose  (inserting the target requires multiple iterations to get alignment correct, so dose is typically higher than when removing devices from the monolith)

- T2 profile and protect monitors checked by Probes Group from the mezannine --> both ok

- Pumping on T2 volume began late morning, leak was found at T2 protect flange (see previous e-log), after correcting this the vacuum pumped down normally.  A helium leak check was performed on all flanges on the T2 monolith --> no leaks found

Oct 9:

- All services attached to target

- T2-MK2 target moved to position 0 using the new cable and the portable control box

- T2 cooling package started (no change to vacuum)

- T2 cooling package expansion tank level checked - the tank was only ~1/3 full, suspected leak through disconnected female hansen connection.  Tank was refilled to normal level using ISIS water

- Plant Group began replacement of blocks over T2 late morning

On Monday Oct 5th a decision was made to uncover and remove the T2 Protect Monitor due to faulty readings.  The monitor had been behaving strangely since the September shutdown, and had been getting worse over time.

- Blocks were removed over T2 starting Monday afternoon.  It is necessary to remove the narrow 12' block and two 6' blocks west of the T2 plug block in order to fit the flask frame in for protect monitor removal

- The monitor was moved to the hot cell by 7pm on Monday.  Pierre was operating the crane and there were no issues with the move.  (monitor measured 15mSv/hr at 0.5m)

Oct 6:

- Monitor inspected: a thin piece of foil was found jammed into the entrance side of the monitor (see photos) & a heat or burn mark was seen on the exit side above and to the south of the plate gap.  It is suspected that this material traveled down the beamline at high speed when there was a vacuum burst during the September shutdown.  In light of this information it was decided to remove the T2 target and T1 target as well (see following e-logs)

- Monitor elevation measured: 1835.8mm (april 2015 measurement was 1836.1 --> ok)

- Old monitor cassette removed, new one installed.  Elevation checked: 1832.2mm - did not match, removed, decontaminated, and returned to Probes for adjustment

- Adjusted monitor installed, elevation checked: 1835.8mm --> ok

- Electrical check done by Probes group at hot cell: found left plate shorted to ground

- Cassette removed, decontaminated, and repaired by Probes Group

- Cassette reinstalled, and electrical check repeated --> ok

Oct 7:

- Monitor elevation re-checked: 1836.2mm --> ok

- T2 protect returned to beamline, bolts installed, cables connected

Oct 8:

- Vacuum pumped on T2 volume after T2-MK2 target returned (see next e-log):  Only reached 2 Torr and a leak was heard coming from T2 protect flange.

- T2 protect was unbolted and lifted ~1in for cleaning of the flange base and o-ring

- The T2 volume then pumped down normally

- A helium leak check of all flanges on the T2 monolith was done by Vacuum Group --> no leaks found

 The following instructions were provided by Graham Waters:

From diag5pc bootup

Password: cfzx29mkdr

Bring up a terminal (click on "terminal icon" left side of screen)
> su
> xpfk34yzjc
> chmod o+rw /dev/ttyS0
> chmod o+rw /dev/ttys1

Bring up terminal emulator GTKTerm
   From booton left of screen
  Click on Applications->Accessories->GTKTerm 

 The T2 protect monitor cassette removed in April 2015 (see E-log #154) was surveyed today.  It was approximately 3mSv/hr at 0.5m and 15mSv/hr on contact.  It is unlikely that this monitor will ever be cool enough for hands-on elevation adjustment that it requires, therefore it has been bagged, labeled, and transferred to the east hot cell for storage.  

Today the T1MK2 target was moved from Position 4 to Position 0 to lower the Rad fields for the 1AM7 repair in the 1A Tunnel. The Rad levels were still to high only dropping by 1000uSv/Hr to 5000uSv/Hr, so the 1AM7 repair is planned for next Tuesday's maintenance day on Oct 6th.  1A will be off till then giving a longer cool down. 

The target was then moved moved back to Position 4 with the Pot ratio reading 0.699. The target was moved using the portable Control box.  The "power on light" on the Control box needs to be replaced.

Note: The MS connector on the 1AT2 cable needs to be tighten before using.  It became loose when swapping the portable Control box cable to the 1AT1 cable.

The cooling package turned on ok after the vacuum was established.

Don Jackson assisted me with this operation and was appriciated....

 The stepper motor that drives the target ladder on T2-MK2 has failed.  This is the cause of the difficulty moving the target ladder yesterday afternoon (see previous e-log).  This was confirmed this morning by attempting to move the T1-MK1 target ladder in the storage pit using the portable control box - the ladder moved with no issues.

The motor is scheduled for replacement during the 2016 winter shutdown unless it is needed more urgently.  Operations have been informed that T2 ladder movement will not be possible until this motor is replaced.

The T1 and T2 targets were moved to position zero, then back to the target position today so that beam profile could be checked using the profile monitors.  Movement for both targets was done with the portable control box.

The T1 target was moved from Position 4 to Position 0, then back to Position 4 (after checking the profile) with no issues.

While moving the T2 target from Position 4 to Position 0 the ladder stopped moving part way through travel.  By toggling the control box "RUN / OFF" switch the motion could be restarted.  This had to be done 2 or 3 times to reach Position 0.  When returning to Position 4 after checking the profile the problem became more severe.  It took over 20 minutes to move from Position 0 to 4 because the ladder only moved slightly or not at all each time the switch was toggled.  This is either due to a failed motor on the target, or failure of the portable control box.

The next step will be to try moving the ladder of one of the target assemblies in the storage pit to determine which is at fault.  In the mean time neither the T1 nor T2 target ladder should be moved.

The warm cell water window tanks were filled on September 17 after 2 weeks of curing time.  A water leak was found coming from the seals of a plastic pipe joining the two tanks near the top.  This was repaired using rubber gaskets and quick cure caulking.  The tank was refilled on September 18 and no longer leaked from that location.  The sand filter was backflushed with the water exhausted to the grass area outside the hot cell lab until the water ran clear.  The pump was then started with the water circulating normally through the filter.  Small water leaks were observed coming from 3 of the 4 plastic ball valves on the filter loop piping.  These leaks stopped after approximately 3 days of continuous running, and are still leak tight now.

During this repair job it was observed that the support wheels for the warm cell manipulators are crumbling and should be replaced before the next time the manipulators are used.

 The RH hot cell lab active sump high level sensor was tested as part of the yearly scheduled inspection.  High water level was simulated by tilting the sensor sideways with a long pole sample scoop.  The Main Control Room and Hua Yang from Nuclear Medicine were notified before the test.  The alarm sounded as expected, and a message was received in the main control room.  This is a latching signal, so a Control Room operator will go to the MHESA RCR lab to reset the alarm.  The water level was approximately 12" below the high level sensor, so the sump may need to be sampled and emptied soon.

Since the previous E-Log the following tasks have been completed:

- Panel spacer rods installed at 4 locations in each tank (two top, two bottom at approximately 1/3 and 2/3 the way across)
- Spacer rods expanded only slightly to begin compressing the rubber window spacers while still leaving at least a 8mm gap between window and frame
- Window and frame around seal area masked off with masking tape
- Sealant was applied from the outside using a plastic cone tip with a copper tube attachment to allow sealant injection to the bottom of the sealant cavity.  The process involved having one person inside the tank to make sure the sealant cavity was completely filled with no voids. A total of 42 of the 20oz. sausages were required.  Two spare unopened sausages remain.

The sealant will now be left for 2 weeks to cure before water is added to the tank.

The following tasks were completed over the past 2 weeks:

- Aquarium Technology Ltd. (UK) was contacted regarding best method for replacing window seal.  A detailed procedure was provided (attached below)
- The warm cell windows were determined to be acrylic, not glass.  Estimated weight: 450lb per panel

- All water in the warm cell windows was released to grassy area outside the lab after approval from Gord Wood (OH&S) and Joe Mildenberger (RPG)
- Warm cell was cleaned up and blocks were configured to make fields as low as possible in the work area around the windows (~2micro Sv/hr max field around windows).  No contamination found on warm cell floor.
- The old seal around each window was cut mechanically with a thin blade
- The panel spacer rods were removed  (most were constructed of SS and Al, however a couple were mild steel and were badly corroded)
- The window panels were separated from the frame using a rubber mallet and moved to the center of the tank (the panels were not removed from the tanks for this repair)
- The old seal material was removed from the panels and the aluminum frame using WD40 or mineral spirits to help break it down (this was a very tedious and unpleasant task)
- Pitting due to corrosion was found in various places on the aluminum frame.  It appeared to be worst where mild steel spacer rods were used, and also worse at weld locations (see attached photo).  Max depth of pits ~1/8"
- The corrosion was cleaned as much as possible using a wire brush then rags with mineral spirits
- A final clean of sealing surfaces was done with isopropyl alcohol
- A deep pit (~1/4" deep) was found on the inside of the inside of the aluminum frame on the west tank, south window (see attached photo)
- This pit, as well as some other smaller ones on the same side were filled with Dow Corning 791 sealant material

- Rubber stand-offs (Digi-Key #SJ5009-0-ND) were installed on the frame side in the center of the sealing faces (3 top, 3 bottom)
- The panels were moved back into place resting on 3/8" thick aluminum spacers (see attached photo).  These replaced spacers of the same size and material that were slightly corroded

The following tasks remain to be completed:

- Install panel spacer rods (mild steel rods to be replaced with SS or Al)
- Apply sealant to all panels from outside the tank as detailed in the procedure using 20oz pneumatic applicator caulk gun (PO: TR206556)
- Install corrosion inhibiting pads inside tank.  Will use 6 pads per tank, (4x McMaster 3609K2 & 2x McMaster 3590K2 per tank)
- Perform final clean of inside of windows and tank
- After 14 days cure time fill tanks with city water and allow to sit to check for leaks
- If leak tight, use water in tanks to back-flush the sand filter
- Refill tanks and start pump with filter in normal mode
- Occasionally check for water leaks and check status of pitting inside tank over the coming months/years
- Also monitor corrosion inhibiting pads and replace or change materials if necessary

Summarized Seal Replacement Procedure provided by Aquarium Technology Ltd:

1) Cut out the viewing panel entirely and mechanically clean all sealant traces from both the panel and the frame bearing faces of the tank structure

2) Degrease all sealing surfaces with something like pure petroleum spirit (mineral spirit) which won't harm the acrylic

3) Reinstall the clean viewing panel spacing it off the frame bearing face by around 10mm using self adhesive rubber stand-offs  (depending on panel flatness likely no more than 3 needed along the top and bottom)

4) Brace the panel in place, mask off the frame and panel, and then inject Dow Corning 791 sealant from the outside.  Ensure that the sealant goes in under pressure, fully wets both surfaces, and completely fills the void.

5) Allow the sealant to cure for 10-14 days before filling the tanks

The 4" Marman flange clamp (IRH0001) labeled "Clamp A",  as used in the testing detailed in Document-114623 Release 1, has been disassembled.  The parts were used to build a TRH1257 clamp which uses the new TRH1259 jaw with 0.480" jaw root width.  This new clamp has been labeled "TRH1257-#01", and will be tested with Helicoflex H-303654 seals in the near future.

T1 and T2 beryllium target failure history was investigated back to 2000 (as far as XSTRIP records were readily available).  For each target cassette the following were noted: length of time the target was subject to direct beam, length of time the target was on the ladder w/o direct beam, and method of failure.  The purpose of the study was to devise a target replacement strategy so that T1 and T2 beryllium targets are preemptively retired from service to reduce the occurrence of failure while running which causes loss of beam time, extra dose to personnel, and damage to equipment.

The following recommendations are the result of the study:
- T1 beryllium targets should be retired after 9.0 months of running time
- T2 beryllium targets should be retired after 12.0 months of running time
- Multiple targets should be installed on each target ladder (at least 2, possibly 3) so that targets may be switched during a running period if required and less target assembly moves are required

A PDF copy of the Excel spreadsheet is attached which includes target use history, failure history, summary of results, and more specific recommendations for each target assembly.  This study and the replacement plan recommendations were reviewed and endorsed by Grant Minor and Yuri Bylinski.  This spreadsheet should be reviewed and updated every shutdown period (including Fall mini-shutdowns), as well as anytime there is a target failure.

- All T1 and T2 values on XTPAGE were stripped back to April 1, 2015:  All systems appear to be functioning normally, nothing unusual observed

- Expansion tank levels checked visually:  T1 level at 1/2" above top sensor nut, T2 level at 1/4" below top sensor nut --> filled to 1/2" above

- Lower air amplifier in use; cool to touch; approximately 25s between each cycle (when air is exhausted from device)

An order for new T1 and T2  beryllium targets was submitted to the TRIUMF machine shop today.  A total of seven T2 targets and twleve T1 targets were ordered to match existing stock of wire EDM target shells.  All required materials were supplied.  Completion by October 30, 2015 was requested.
 

A complete set of new drawings for both targets were released in June 2015 (TBP1683 for 12mm T1 Be target; TBP1691 for 10cm T2 Be target)

Current target stock for T1 (12mm Be) includes: 1 target on the shelf, 1 target on the MK1 target assembly (position 4, has not seen beam), 1 target on the MK2 target assembly (position 4, currently in use).

Current target stock for T2 (10cm Be) includes: 1 target on the shelf, 2 targets on the MK1 target assembly (position 3, has seen beam;  position 2 has not seen beam), 1 target on the MK2 target assembly (position 4, currently in use).

 A 5lb roll of 347SS sheet was received today.  This alloy is to be used for the target windows of the new T1 and T2 beryllium target cassettes that will be manufactured in the near future.  The material was ordered from Alloys International (PO# TR206407).  A mill spec report was included with the material, a scanned copy is attached in PDF format.  The roll will be stored in the meson hall RH hot cell lab office on the shelving above the work bench.

 The water in the BL1A holding tank was released to the city sewer.  This water was sampled, tested, and approved for release by RPG two weeks ago.  The total volume was 510L, a combination of water from the T2 and TNF systems.  The total time required to pump out the water with all valves fully open was 1hr2m.  Paperwork for the water release was completed and returned to RPG.

The new T1/T2 level sensors (Omega LVCN414) were hooked up to the PLC and the test performed on June 11 was repeated.

The water level above the base of the bucket was measured using a tape measure, as well as with the level sensor at various intervals as the bucket was filled and then drained:

Tape Measure Level (cm) --- Reading from Ultrasonic Level Sensor (cm)

Empty --- 0
3.3 --- 3.3
6.7 --- 6.7
12.2 --- 12.2
20.5 --- 20.3
29.2 --- 28.5
36.6 --- 36.3  (Last measurement on June 17, left overnight, see graph below)
36.6 --- 35.6 (First measurement on June 18)
31.3 --- 30.2
22.0 --- 21.9
13.1 --- 12.3
7.3 --- 6.5
Empty --- 0

Data was recorded while the sensor was left overnight (see attached plot).  After approximately 2 hours, the sensor stabilized and then fluctuated less than 0.1cm for the next 12 hours.  Overall these tests proved that this sensor will be suitable for the intended application and the parameters used for configuration are appropriate.

The new T1/T2 level sensors (Omega LVCN414) were tested in the RH HC lab today using the Omega software on a laptop and positioning the level sensor flange above a bucket of water with similar height as the expansion tank.
The level sensors behaved as expected with varying water level.

The level sensor was configured with the following parameters:

- Loop Fail-Safe: Empty
- Output at Empty: 4mA at Bottom
- Start-up condition: Empty
- Sensor Height: 50.1cm (Based on TBP1682)
- Fill Height: 45.0cm (Based on TBP1682)

This results in 4mA output at 0cm tank height, and 20mA output at 45.0cm tank height.

This test will be repeated next week with the sensor wired directly to the PLC.

At 9:15am this morning BL1A tripped off due to loss of the watchdog signals for the T1/T2 control system.  Graham Waters investigated and the problem was found to be a corrupted initialization file.  The error was corrected, and BL1A was started by 11:30am.
 

Grant moved the T1 and T2 targets from position 0 to position 4 this evening at around 9pm.

The position 4 selector button on the T1 rack was not working.  The manual control box had to be used with the adapter cable and military connectors for both T1 and T2 to achieve position 4 on both targets.

Replacement of the BL1A blocks was completed yesterday (May 8, 2015) in preparation for first beam down 1A on May 12th.  The attached PDF shows a photograph of the current arrangement, and also the block arrangement at the beginning of the 2014 running period for comparison.

The Meson Hall RH Lab Warm Cell was used by Beamlines Group for hydrostatic testing of the TNF heat exchanger.  The location was chosen because of access to a water supply, active drain, and the heat exchanger has removable contamination.  The work took place on April 30th and May 1, 2015.  New plastic sheet was placed over the 6 foot block in the center of the warm cell which was used as a work area.  The plastic was removed after completion of the job.  The floor around the block was swiped and 100cpm contamination was found.

See attached photos.

 The T1-MK1 and T2-MK1 targets have been re-wired to allow version identification.  Work was performed in the target storage pit.  Approximate field around the targets in the pit was 100microSv/hr.  Disposable coveralls were worn for the job and had 800cpm of contamination on completion.  All nylon cable ties on the targets were replaced with PEEK cable ties.  Total dose received was 0.06mSv by Isaac Earle.  This modification has now been completed for all T1 and T2 target assemblies.

T2-MK2 target information sheet, beam spot measurements, and elevations were updated and delivered to Operations. PDF copy attached.

No work was done to T1-MK2 (placed in storage pit for 2015 shutdown duration to lower fields), so the sheets will not be updated.

An electrical check has been completed for the T1 & T2 Profile and Protect Monitors by Dave Cameron and Mike Russell.  They report that all results look normal.

- T2 Protect Monitor moved from the beam line to the hot cell on Monday April 20th
- Elevation of the plate center measured at 1839.0mm from the base of the vacuum flange
- Old monitor cassette photographed (attached).  Heat mark noticed on exit side approximately 10mm above and 5mm south of plate centers (cause unknown, and no matching mark on entrance side)
- Old monitor cassette removed, new one installed.  New elevation 1836.1mm (2.9mm upward shift - 3.0mm was requested)
- Electrical check performed by Probes Group on April 22 - They reported that it looks normal
- T2 Protect Monitor returned to the beamline April 22 in the evening.  Cables plugged in and flange bolts installed
- Vacuum pumped down normally on April 23 in the morning
- Probes group repeated the electrical check after installation from the 1A mezzanine - They reported that it looks normal
- The old monitor cassette will be left in the hot cell for possible future use

 The T1-MK2 target was wired for automatic target version detection by the new PLC control system (A to C jumper  in POT/CAM MS connector).  T1-MK1 and T2-MK1 targets still need to be wired which can take place after the shutdown without moving them from the storage pit.

Both cooling packages are running steadily with the targets in Position 0.  

The demin flow (Q2) for both was tuned to approximately 1.5USgpm.  

A defeat was applied to T2Q1 flow to prevent high level nuisance trips.  This will remain in effect for the 2015 running period.

T2P1 Sensor (water inlet pressure) has a noisy signal.  This was not improved by replacing the sensor.  It is not expected to cause issues during operation and will be addressed (if still present) after the 2016 upgrade to PLC/EPICS control.

The expansion tank for both packages was topped up to approximately 1/4" above the top nut of the warning level sensor.

 Four water sensors have been installed on top of the T1 monolith.  These water sensors were wired and brought to the area during the 2014 shutdown, but installation was delayed so that the top of the monolith could be painted first.  The sensor labels (attached to the sensor cable) are: T1, T2, T6, and T8.  The location of each sensor is documented on the diagram below.  Two photos of the top of the monolith are also attached.  Doug Preddy has been notified.

 The lower air amplifier has been started, and air supplied to the T1 and T2 areas by opening the appropriate ball valves beneath the amplifier.  The amplifier is cycling normally and the barrels are cool to the touch (indicating no significant air leaks).

Actuation of both profile monitors was checked from the control room.  Both reach the in and out limits as expected, with no change in vacuum level in the respective volumes.

 The T1-MK2 target was transported from the storage pit to the beamline.  The target measured approximately 10mSv/hr at 0.5m.  All target services were hooked up and the cooling package was started (no change in T1 vacuum).

 The T2-MK2 target was moved from the hot cell to the beam line. The T2 Protect Monitor was moved from the beam line to the hot cell.  The T2-MK2 assembly measured 136mSv/hr at 0.5m, and the protect monitor was 8mSv/hr.  Total dose incurred by Isaac Earle:  0.18mSv.  No dose to other workers

The "POT/CAM" MS connector on the T2-MK2 has been wired to allow automatic target version number identification by the new T1/T2 control system.  Pin A was jumpered to Pin C.  (MK1 targets will use Pin A to Pin B jumper).

The nylon cable ties on the target assembly were found to be very brittle and fragile, they have been replaced with PEEK cable ties.

The T2-MK2 target is now ready to be moved to the beamline.

The Inlet (T1), Outlet (T2), Col.A (T6), and Col. B (T7) thermistors at the T2 cooling package all gave intermittent faulty readings at various times between Oct - Dec 2014.  It is suspected that this was caused by a water leak in the area causing the terminals to get wet.  Inspection of the sensors revealed that they are in poor shape and that the readouts sometimes fluctuate when the probe or connectors are handled.  All four thermistor probes have been replaced.  The plastic interior of the Lemo connectors were found to be crumbling, and were replaced with crimp connections.  All T2 thermistors now appear to be functioning normally.  The T2 cooling package was started (with a bypass instead of the target assembly).  Flow through Collimators A and B has also been started.  The T2 package was inspected while running and no leaks were found.

Cyclotron Fault #7722 has been returned.

Elevation measurements of the T2-MK2 target were taken in the hot cell to confirm previous records in preparation for the T2 Protect Monitor adjustment that will take place later this shutdown.  Potentiometer readings were also checked for ladder position 2mm above and below nominal in case we decide to run the target in an offset position again.

Entrance Side:

Ladder moved until potentiometer at 16.19kΩ (nominal position 4)
- Measurement on tape: 1838.7mm

At elevation 1836.7mm: 16.04kΩ
At elevation 1840.7mm: 16.27kΩ

Exit Side:

Ladder moved until potentiometer at 16.19kΩ
- Measurement on tape: 1837.9mm

At elevation 1835.9mm: 16.04kΩ
At elevation 1839.9mm: 16.29kΩ

The elevation measurements at nominal position agree with those taken in May 2014.
The T2-MK2 target is now ready to be returned to the beam line after completion of the M9-T2 work.

Keith C. Ng and Isaac Earle both received .02mSv dose during this procedure.

The following work was done today at the T2 cooling package:

- Remainder of #4 Hansen o-rings replaced  (see E-log from Jan 30, 2015)
- Installed the following ball valves which have been rebuilt & tested with 100psi air: target outlet 3/4" (x2), resin can outlet filter isolation 1/2" (x2), heat exchanger secondary side outlet 3/4" (x2), collimator A outlet 3/4", collimator B outlet 3/8"
- Installed the following ball valve which has been tightened & tested with 100psi air (no rebuild kit available): resin supply side 1/2"
- Disassembled the target water inlet thermistor probe assembly to check compatibility with thermocouple probes purchased for T1/T2 controls upgrade
- Misplaced a fitting that the probe inserts through, plugged thermistor port as a temporary measure (the outlet thermistor still functions as machine protection)
- Filled cooling package and opened CuALCW supply and return valves to T2 area in BL1A tunnel
- Started package, inspected for leaks, tightened various Swagelok fittings to stop minor leaks
- Tightened valve stem packing compression nut on Col.A/M8BB shut-off valve and Col B. shut-off valve on cooling package front panel to stop drip leaks from the valve stems
- Inspected again for leaks while cooling package running:  none observed

The T2 cooling package is ready for a interlock response timing test which will be performed tomorrow (part of T1/T2 controls upgrade).
Before the T2 cooling package is ready for operation, the target water inlet thermistor must be reinstalled, and the thermistor connectors must be replaced (if the thermistors are not replaced with thermocouples as part of the controls upgrade)

Total dose:

Isaac - 0.08mSv
Keith - 0.01mSv
 

 Ron Kuramoto, Keith Ng, John McKinnon, and myself spent some time today investigating water and air lines related to the T1 Rotary Collimator (Col. A).  We discovered the following:

- There is a panel on the back (south) side of the T1 cooling package with 3 air lines for Col. A actuation, Col A cooling water supply & return, and the Col. Shield water cooling supply and return
- The air lines coming from the panel were decomissioned in 2003 by Tom Lyth, and at least 2 out of 3 were replaced with CuALCW water from the M13 header in the BL1A tunnel
- Motion / actuation of the rotary collimator was disabled in Oct 1994 and it is now used only in one position
- The two water lines coming from the M13 header (which connect to the actuation cylinder) will be disconnected, capped, and tagged.  The M13 header can then be completely removed
- It is suspected, but not yet 100% confirmed that Col. A and the Collimator Shield are still cooled by the lines passing through the panel at the back of the cooling package
- A flow test will be performed to confirm this, which will also confirm that the Q4 flow meter at the cooling package measures cooling water flow to Col. A

- The T1-MK1 target was moved from the hot cell to storage pit hole #3 (ready as T1 spare)

- The T2-MK2 target was moved from the beam line to the hot cell for inspection and in preparation for T2/M9 work this shutdown
- The target measured up to 90mSv/hr at approximately 1m while being lifted
- Beam spots on the target will be measured and recorded to determine ideal offset (if any) for next running period
- A 10mSv/hr collimated field was measured coming from the T2 monolith hole
- An aluminum blank-off was installed over the hole followed by a stack of circular steel discs for shielding which reduced the max field in the area to 700uSv/hr

This is a continuation of work detailed in E-Log #129 and #140.

- Static water test performed on the target cooling lines:
     - Test setup assembled and leak checked by pressurising with house air, submerging in water and checking for bubbles (see photos)
     - System filled, pressurized, and valved off on Friday Feb 13 at 5:30pm,  Gauge read 60psi
     - Checked Monday Feb 16 @ 9:00am.  Gauge read 56psi
     - Checked Monday Feb 16 @ 5:00pm.  Gauge read 56psi
     - Checked Tuesday Feb 17 @ 9:30am.  Gauge read 55.5psi  (0.5psi / 24hrs rate)

- Checked results with Dimo and he requested a helium leak check be performed with a newer model leak detector to quantify leak rate:
     - Using Varian 979 leak detector, pumped ~1.5hrs w/ cold trap to remove water
     - TP press: 0.0*10^-4 torr,  Base leak rate ~7.0*10^-9 atm cc / sec
     - Tested all normal leak check locations and did general helium flood with no response at all  (1.5psi helium pressure)
     - Over ~ 20 minute period the base leak rate gradually rose to 8.8*10^-9 atm cc / sec, at one point spiked to high 10^-9 level, then returned to base line
     - Reviewed results with Dimo, he requested test be repeated after the target sits vented overnight

- Target vented and left overnight, leak check repeated Feb 18 starting 9am:
     - 0.0*10^-4 torr TP pressure after ~5 mins
     - 10^-9 atm cc / sec leak rate level after ~20 mins
     - All locations sprayed with 1.5psi helium, general flood in ladder area --> no response (Dimo and Edi present)
     - Target deemed OK for use in beam line by Dimo, he requested the cooling lines be filled with helium after installation in b/l while leak checking T1 volume

- Target will be transported to the storage pit tomorrow (Feb 19) and is ready for use as the spare T1 target

The target had an obvious helium leak when first tested (E-Log #129 and #140), which could not be found when helium leak checking again after the static water test.  Some possible explanations are:  water remaining in the lines blocked the leak path,  water remaining in the lines froze when vacuum was pulled blocking the leak path, having water in the lines caused some corrosion or other deposit which plugged the leak.

The helium leak at plugged position 1 right side measured before the static water test is the same, or slightly less than the leak rate when the target was first removed from the beam line.  There were no T1 vacuum issues during the running period, therefore no issues are expected when this target is put in service again.