On May 23, 2024 two sets of plugs from position 1 and 3 and a 12mm Be target from position 5 (serial #109) was removed from T1-MK1 in the Meson Hall remote handling hot cell.

The plugs and spent target are kept in the secondary hot cell (East) on the lift table for future beamspot imaging and to cool down prior to disposal.

In their place 12mm Be targets were placed:

• #107 in position 5
• #110 in position 3
• #111 in position 1

We will continue with leak checking and position measurements in the coming week, at which point this ELOG will be updated.

------------------------------------------------------

UPDATE May 24:

• leak testing began on the T1-MK1, vacuum stagnated at 1.5e-2 Torr and a leak rate of 3.9e-07 atm-cc/sec (system vented, will re-tighten target fittings and re-do leak test). 
• after tightening and repeating pump-down in the afternoon, the vacuum volume stalled again at 1.5e-2 Torr. releasing helium to the target fittings registered a response in the leak tester at all virtually locations (for either side of the ladder). 
• the plan for the coming monday is to re-check the vacuum fittings and re-tighten the swage-lok connections to the targets before repeating the leak test once more.

------------------------------------------------------

UPDATE May 27:

• the swage fittings for the targets on position 1,3,5 were tightened in the morning but the volume still couldn't be pumped down to 1e-03 Torr.
• pumping down only on the vacuum line (up to KF elbow ahead of hansen fitting at outlet of target connections pannel) allowed the pressure to drop to 1e-03 Torr so it was determined that the leak tester + associated vacuum connections were not the problem.
• shortly afterwards, we troubleshooted by pumping only to the 'high pressure test' setting (no Turbo pump) and sprayed helium to various joints above the target flange. we learned that there is likely some leaking through the hansen fittings + the swage elbow from the fittings down through the target flange.
• plastic ferrules are used in this elbow to allow for some seals to be replaced, so this may be the source of the leak.
• then, the fittings on the target ladder were re-tightened and helium was sprayed onto various fittings on the ladder. all but the left join on target ladder #1 (position 9a) does not register a leak rate above baseline after tightening.
• at end of day, the ladder was able to consistently reach a base-line leak rate of 1.7e-07 Torr-l/sec and a vacuum of 4e-3 Torr.
• we will proceed with replacing the water line connections on top of the target flange and re-check the leak-tightness of the ladder in the coming days.

------------------------------------------------------

UPDATE June 14:

several attempts at replacing the plastic (Nylon, Swage NY-1013-1 and NY-1014-1) ferrules and re-establishing the fittings on the target service panel were made since the last update. it was determined that the lines leading from the leak detector to the target panel was leak tight (reached 1e-03 Torr within ~5 minutes of pumping down). a double-male Hansen adapter was made to test this (see image). in the final attempt to re-seal, the nylon ferrul-ed fittings were tightened last and the target managed to pump down to 2e-03 Torr, with a stable baseline leak rate of ~2e-08 Torr-L/s. note that prior to exchanging targets, the leak rate obtained during water flushing was 1e-10 Torr-L/sec. additionally, spraying helium on the inlet fittings still registered a response at the leak detector (~4e-6 Torr-L/sec in the worst case).

we think that the following happened to the target, leading to the leak: at some point the target panel collided with something by the water outlet hansen port (see image), this caused the nylon ferrule at that line to crack/become damaged but no leaks occurred immediately after. when we performed the target exchange, the water line was disturbed in some way either as a result of torque-ing the target fittings or when the ladder was moved on the turntable causing the joint to completely fail and leak. because of the damage from the collision, after taking the fittings apart, it is no longer possible to return the fittings and for them to seal.

the plan moving forward is to implement some minor design changes to 1) replace the damaged fittings on the target panel, 2) better facilitate replacement of the nylon ferrules in the future, 3) better facilitate leak testing on the target service panel connections.

------------------------------------------------------

UPDATE Sep 27:

• After designing, releasing, and procuring the leak fix components (see TRH1681), we implemented changes to the electrical barrel connectors and tested the fit of the cooling line piping (see pictures).
• We checked that the ladder motor, potentiometer, and limit switch circuits all worked properly.
• Some differences were noted:
  • D10414-2 accepts 1/4"-20 screws to mount the panel, not 10-24 like shown in the legacy drawings, the clearance hole on TRH1682 had to be adjusted accordingly.
  • The low profile screws specified for the barrel connectors were difficult to use (item 108 in TRH1681), so instead we used normal 4-40 socket head cap screws and were able to check that the barrel connectors could fully couple without interference from the socket head.
  • We did not use lock washers for the barrel connector flanges because they drop too easily and instead we elected to tighten the nuts well.
  • Some loose leads fell out of the potentiometer (bottom left) and profile monitor (bottom right) barrel connectors. These were not electrically connected to anything so we decided to remove them, they will be kept in a recorded location at the end of the operation. 
• In the coming week, we will look to fasten the water line fully and check that they are leak tight.

 ------------------------------------------------------

UPDATE Oct 02:

• The water line was fastened fully, some items noted:
  • Order of fittings tightened: elbow to bulkhead adapter > elbow to reducer > 5/8" connector to tube > elbow with reducer to tube > nylon ferrule connection
  • We tried to tighten the elbow with reducer to the tubes with the tubes in position to minimize loading the nylon ferrule at the end but found that the outlet tube still ended up protruding significantly from the bulkhead plate (see image). For future operations we should make a note to not load the outlet tube too significantly because it is only supported by the tube (bulkhead nut not fully engaged).
  • The above issue likely resulted from bending the tubes to the 3D printed template but yielding misaligned ends. In the future better templates that emphasize absolute end alignment for tube bending should be made
  • The NPT-Hansen nipple was not tightened before the water line was put on which may have applied unnecessary loads onto the tubes. In the future we will make a note in the drawing on which fittings to tighten first and how to perform the operation to minimize loading on the nylon ferrules
  • In the future some modifications can be made to help relieve this issue for version 2 of the water connections, maybe using flexible metal tubes or thinner walled pipe, alternatively we can implement a more flexible mounting approach for the Hansen ends.
• We noted some tight clearances on the electrical bulkhead. In future implementations of the fix, the connectors can be spaced farther apart or staggered, also they should be moved further down from the top flange of the target ladder.
• Leak testing proceeded with the following results:
  • Total time under vacuum ~2 hours
  • Reached 'fine' vacuum within 1 minute, pressure saturated to 2e-3 Torr within 3 mins 30 secs.
  • Baseline leak rate of 0.0e-10 Torr-L/s reached within 7 minutes of pumping down.
  • ~1.4e-9 Torr-L/s peak @ 15s delay (to first signal) after 1s x 3psi He to general vicinity of newly made water line connections << deemed acceptable
  • 0.5s x 3 psi bursts of He
    • ~0.9e-09 Torr-L/s peak @ 15s delay to nylon ferrule outlet line (did not change after further tightening) < deemed acceptable
    • ~0.2e-10 Torr-L/s peak @ 15s delay to nylon ferrule inlet line < deemed acceptable
    • ~0.5e-10 Torr-L/s peak @ 40s delay to NPT-Hansen outlet < deemed acceptable
    • ~0.4e-10 Torr-L/s peak @ 40s delay to NPT-Hansen inlet < deemed acceptable
  • SS swagelok connections not tested thoroughly due to low likelihood of being a leak source (all tightened properly to 1.25 turns as per swagelok directions)
  • He 'bleed' test by closing hot-cell He wand solenoid valve and bringing wand opening ~5 cm from pos.5 target swagelok resulted in a stable leak rate of 0.8e-8 Torr-L/s << target ladder leak likely prevents pressure in target ladder from bottoming out.
• Will carry on with leak testing the target ladder and possibly the swagelok fittings on top of the hot cell if necessary by the end of the week.

 ------------------------------------------------------

UPDATE Oct 04:

• The target ladder was pumped down at ~11 am and reached the same baseline as yesterday (0e-10 Torr-L/s @ 2e-3 Torr within 3mins, 30 sec), but in the afternoon, the baseline leak rate worsened to 1.0e-8 Torr-L/s.
• For reference, 0.5s of 3psi He was delivered to the water outlet nylon ferrule joint and registered a response of 1.8e-8 Torr-L/s with a delay of 20 sec. A 3s spray at 3 psi around the water fittings registered a 4.9e-8 Torr-L/s response with a 15s delay.
• The following was observed from sending 0.5s of He to the target ladder Swagelok fittings (see Img for more details):
  • Leak rate baseline improved during testing (~over 1 hour) to 3.0e-9 Torr-L/s
  • Bellows, and targets 2-5 registered leak rates in the order of 1e-8 Torr-L/s
  • Target 1's left Swagelok fitting registered the highest leak rate at 2.8e-7 Torr-L/s (when the leak rate baseline would have dropped to the lowest value of 3.0e-9 Torr-L/s)
• We checked the KF and hansen fittings to make sure that they are not a leak source and managed to bottom out on both the leak rate and pressure readings on the leak tester.
• We will take off Target 1 in the next session, run a die over the male threads, clean the conical face of the male thread with scotchbrite,and attempt to re-do the target joint before leak testing once more.

 ------------------------------------------------------

UPDATE Oct 11:

• We removed target at pos 1 (ID 111), noticing that the swagelok nuts were easy to undo.
• Working theories on how they became 'loose' include: 1) creep undoing misalignment between the two swagelok tubes on target, 2) vibration from the nuclear ventilation onto the target ladder undoing the joint, 3) thermal changes between summer and 'fall'.
• The threads on pos1 seem worn, it may be a good idea to plug this ladder position permanently to avoid further damaging the threads and jeopardizing the ladder in the future (see image).
• The ferrules on the target seem to be in good condition but we should still clean them with the scotch brite tool
• We added plugs over the exposed male swagelok ports to protect from debris entering the water line.
• We placed the removed target between two wypalls and weighed the wypall down with a wrench to protect it from debris.
• The 3D printed scotch brite tools need to be re-designed for 3/8" drive size (was designed for 1/2" drives).
• We will return to bag, and package the target for safe keeping in the secondary hot cell; clean the male swagelok threads and sealing faces on pos 1 target ladder; and possibly place permanent plugs in pos 1.

 ------------------------------------------------------

UPDATE Oct 15:

• The leak tester was calibrated with an external calibrator. Original leak rate 1.3e-7 Torr-L/s, 2% loss over ~10 years >> ~1.1 Torr-L/s leak rate.
• The leak checker registered an initial leak rate of ~2.3 Torr-L/s before calibration so we corrected the measurement (at least within the calibration range) by a factor of 2.
• This voids old leak testing data that we made on the target ladder.
• We will add a note to the procedure to ensure that the leak detector has been calibrated before performing leak testing in the future

 ------------------------------------------------------

UPDATE Oct 23:

• After some testing with 3/8" tube Swagelok threads outside of the hot cell, we determined that the reason the threads in pos 1 of the target ladder was 'difficult' to run was likely because the nuts were over-tightened previously.
• There is a risk of compromising the threads if we run the die over over-tightened threads because they have been shifted from where they should be. We will instead attempt to replace the plugs, after cleaning the sealing face on the male threads using the cleaning tools.
• We also checked with a Swagelok tightening gage (3/8) to see whether the other targets were tightened properly, and all had gaps smaller than the gage. Note that we could not get fully into the gap with the gage on the side by the profile/protect monitor due to interference with a support column with the gripper. We will make an extender tool to avoid this issue in the future.
• We also bagged the target that was previously mounted onto pos 1 on the ladder and moved it to the secondary hot cell for storage (temporary).

 ------------------------------------------------------

UPDATE Oct 24:

• The threads and sealing face on pos 1 were cleaned with scotch brite, blown with compressed air from the inside by connecting compressed air line at the service panel, blown with compressed air from a bottle externally, and wiped with a wypall with the manipulators.
• The old plugs were moved to the secondary hot cell, new plugs were tightened onto pos 1.
• Initially the baseline leak rate was 3.3e-8 Torr-l/s at 4e-3 Torr. With He, we were able to determine that the plugs, specifically position 9a, was the most leaky joint in the ladder.
• After a couple rounds of tightening the plugs, and making sure that the remaining targets were also tight, the baseline improved to 1.7e-9 Torr-l/sec. The plug at position 9a also no longer registered a leak response from having the He line brought up to the joint.
• The plug at position 9a still is the leakiest location in the ladder however, registering a peak leak rate of 2e-8 Torr-l/s at the final baseline.
• We deem that the ladder is sufficiently leak tight, and we will move on to replacing a rubber coupling on top of the target ladder before performing target position measurements.

 ------------------------------------------------------

UPDATE Nov 25:

• The coupler was successfully removed with a custom puller tool (see TRH1730, TRH1731, TRH1732, and Document-244878). This tool is used for the 3/8" shaft side.
• The press fit coupler had to be cut up with a dremel and saw before using the tool.
• The old coupler had to be 'cut-up' to be removed. During initial troubleshooting, the rubber body split, indicating that it was near end of life.
• The replacement coupler system could be inserted without removing any parts.
• The relative alignment of the shafts may have changed up to 1/2 a turn during this process (the printed clamps were not strong enough to resist torques exerted when attempting to remove the press fit coupler).
• Potentiometer table will need to be reviewed when performing target measurements.
• Light loctite to be added to the treads.
• Note: the old coupler has stainless shaft adapter (3/8" step down).

 ------------------------------------------------------

UPDATE Nov 28:

• Loctite added to all coupler screws (including flexible coupling from motor to three-way gearbox)
• Final leak check completed on replacement piping on service panel with the following results
  • baseline leak rate bottomed to 0.0e-10 Torr-L/sec within 7 mins and pressure bottomed to 1e-3 Torr within 5 mins.
  • baseline leak rate worsened to around 1e-9 Torr-L/sec by the end of leak testing, assumed due to increase in amount of trapped He in the line through testing
  • (time to detect [s] || peak leak rate [Torr-L/sec]) values for 3psi, 0.5s Helium using wand on top of hot cell
    • inlet side:
      • 20s || 3.8e-10 for nylon ferrule joint
      • 20s || 6.3e-10 for o-ring flange
      • 20s || 2.7e-10 for hansen NPT thread
    • outlet side:
      • 15s || 3.5e-10 for plastic ferrule joint
      • 15s || 5.1e-10 for o-ring flange
      • 20s || 0.8e-9 for hansen NPT thread

The target ladder was moved to the lift table in preparation for measurements.

 ------------------------------------------------------

UPDATE Jan 03, 2025:

• Measurements (elevations) of the targets and alignment jig were completed over December to early January.
• Notes on measurement procedure:
  • Used the old alignment jig instead of new (easier to insert/remove)
  • Noticed that the target ladder potentiometer is jumpy at around 12 kOhm
  • Placed the jig between positions 4-5 throughout (pot reading of 11.65 kOhm places the jig in beam height - in line with protect monitor)
  • For left-right measurements, moved the alignment jig 'into-position' (i.e. set potentiometer value to 11.65 kOhm) to 'zero' the micrometer before changing ladder position (potentiometer value) back to the target being measured (turntable height not touched after zero-ing off to the jig)
    • For the entry side, because the protect monitor is obstructing, target 5 had to be measured above the protect monitor (jig also zeroed off above the protect monitor, i.e. not at the resistance value given above), while all other targets were measured below.
    • This introduces some error since we are measuring the target not in the beam position.
  • Target height measurements were done with measuring tape as normal.
  • Targets were measured with turntable flange having an angle of ~0.2 degrees, plumb bob and jig measurements taken with turntable flange 'leveled' as best as possible (angle of ~0.02 degrees)
    • For future measurements, the turntable should be also leveled before target measurements.
  • Levelling the turntable flange was done with a Digi-pass digital level (DW-1300XY, in absolute level mode, calibrated before starting)
  • By the cutout, the turntable flange is less stiff, causing level measurements to be tilted inward. We therefore relied more on level measurements on solid portions of the turntable flange as well as on the target flange.
  • Levelling bolts could be turned with target ladder seated.
  • If the turntable is not rotated, and only raised/lowered, the levelness of the table and target remained approximately constant (e.g. from 0.02 degrees change to 0.04 degrees as the turntable is lowered fully from fully raised).
  • If rotated, the turntable level changes significantly (e.g. from 0.02 degrees to 0.15 degrees)
  • The flange was re-levelled between entry side and exit side measurements for the plumb bob and alignment jig measurements
  • With new nuclear ventilation filters, flow inside the hot cell perturbs the plumb bobs and prevents them from settling.
  • Nuclear ventilation had to be turned off momentarily for the plumb bob measurements
  • Nuclear ventilation is returned when working on the hot cell roof
  • Nuclear ventilation is controlled by turning the toggle on the electrical disconnect panel from 'hand' to 'off' ('auto' setting assumed not used)
  • Respirator and lab coat worn throughout plumb bob measurement job, air survey done by RPG after job (confirmed OK)
  • Area also 'taped off' during plumb bob measurements to prevent non-worker entry
• The coupler exchange resulted in around a 0.4mm shift in the target heights
• A shoulder screw and wood clamp is used to lock the plumb bob jig in place (old screw/nut missing, no other components in toolbox provides locating fit for pivot joint on jig)
• Assumed new plumb bob measurements more accurate due to better/more accurate 'levelling' equipment (digital level).

 Between Nov 14 ~ Dec 9 the following activities took place:

- The leaky 10mm Be target at pos'n 3 was removed and plugged
- The loose protect monitor electrical conduit was secured in place using an aluminum shim
- David Cameron performed an electrical check on the protect and profile monitors and got the expected response from both
- The target was flushed with water, then air
- The target was pumped down using a cold trap to capture remaining moisture
- The target reached a minimum pressure of 80mTorr on the leak detector Hastings gauge
- Helium leak check revealed a leak at both of the position 1 Swagelok caps (up to 80 on 50x scale on left side, and 100 on 100x scale on right side when facing ladder with 1s spray of 5psi helium)
- The leak rate did not improve with tightening of the caps
- The caps were removed, ladder side threads inspected (no obvious damage observed), and new caps installed
- Helium leak check repeated: Pos'n 1 left side now leak tight, right side leak rate was worse (could not completely open throttle valve on leak detector)
- Torque was increased on the cap with no improvement in leak rate
- The right side cap was removed and when trying a new cap it would not spin freely
- The ladder side threads were inspected more carefully with the Nikon level:  a small dent across the first thread at approximately 2 o'clock position was seen as well as some material build-up or possible galling seen at the 1st and 2nd threads around the 4 o'clock position.
- The threads were filed to improve their profile which allowed a new cap to be installed with light resistance (still would not spin freely)
- Leak check was performed with a minimum pressure of 70mTorr reached (throttle valve fully open, roughing closed), with large response still at the right side pos'n 1 cap, No change with moderate tightening

It is suspected that the ladder side fitting sealing face is damaged causing a poor seal.  When a cap was installed and torqued this may have caused the galling or material build up observed at the 4 o'clock position (but not the dent on the 1st thread at the 2 o'clock position)

Various experts on-site will be consulted about how to best proceed before doing further work.  One option is to perform a static water test.  If the target is water leak tight then it may be used in the beam line.  If repair efforts are not successful and the target is not water leak tight then the target ladder will have to be replaced which is approximately a 2 week job.

 The T1-MK1 target is currently in the hot cell for repair of a leak at position 1.  See E-Log #129 for details on previous work.

- Pos'n 1 right nut removed (a little stiff, but no excessive force required)
- Exterior threads cleaned w/ Scotchbrite disc on Dremel tool
- Inside sealing face inspected w/ Nikon level.  Possible brownish material on sealing face at 10 o'clock and 12/1 o'clock positions
- Threads inspected w/ Nikon level: small dent at 1st thread 2 o'clock position as seen before and small amount of material build-up or galling at 1st & 2nd thread 4 o'clock position.  Otherwise threads looked clean and straight
- Constructed Dremel attachment using ~1/2" thick, ~4" long piece of Scotchbrite folded over a 1/16" diameter steel rod, secured with zap-straps and shaped w/ scissors to fit inside the ladder port (see photo)
- Cleaned and polished the sealing surface w/ new Scotchbrite Dremel tool
- Inspected inside sealing surface: brown coloured material no longer visible.  A clear scratch/dent is visible at 10 o'clock position.
- Blew out hole and new nut with compressed air
- Installed nut (went on much easier than before which indicates that the Scotchbrite disc thread cleaning method is effective)
- Leak check performed:  pumped down to ~70mTorr on Hastings gauge w/ throttle valve fully open and roughing valve closed
- Sizeable leak at Pos'n 1 right side nut, as before.  No improvement with tightening

The next step will be to perform a static water test on the cooling lines to see if the plug is water leak tight.  If it is, no further action is necessary.  If not, either the ladder must be replaced, or other methods to achieve a seal investigated.
 

This is a continuation of work detailed in E-Log #129 and #140.

- Static water test performed on the target cooling lines:
     - Test setup assembled and leak checked by pressurising with house air, submerging in water and checking for bubbles (see photos)
     - System filled, pressurized, and valved off on Friday Feb 13 at 5:30pm,  Gauge read 60psi
     - Checked Monday Feb 16 @ 9:00am.  Gauge read 56psi
     - Checked Monday Feb 16 @ 5:00pm.  Gauge read 56psi
     - Checked Tuesday Feb 17 @ 9:30am.  Gauge read 55.5psi  (0.5psi / 24hrs rate)

- Checked results with Dimo and he requested a helium leak check be performed with a newer model leak detector to quantify leak rate:
     - Using Varian 979 leak detector, pumped ~1.5hrs w/ cold trap to remove water
     - TP press: 0.0*10^-4 torr,  Base leak rate ~7.0*10^-9 atm cc / sec
     - Tested all normal leak check locations and did general helium flood with no response at all  (1.5psi helium pressure)
     - Over ~ 20 minute period the base leak rate gradually rose to 8.8*10^-9 atm cc / sec, at one point spiked to high 10^-9 level, then returned to base line
     - Reviewed results with Dimo, he requested test be repeated after the target sits vented overnight

- Target vented and left overnight, leak check repeated Feb 18 starting 9am:
     - 0.0*10^-4 torr TP pressure after ~5 mins
     - 10^-9 atm cc / sec leak rate level after ~20 mins
     - All locations sprayed with 1.5psi helium, general flood in ladder area --> no response (Dimo and Edi present)
     - Target deemed OK for use in beam line by Dimo, he requested the cooling lines be filled with helium after installation in b/l while leak checking T1 volume

- Target will be transported to the storage pit tomorrow (Feb 19) and is ready for use as the spare T1 target

The target had an obvious helium leak when first tested (E-Log #129 and #140), which could not be found when helium leak checking again after the static water test.  Some possible explanations are:  water remaining in the lines blocked the leak path,  water remaining in the lines froze when vacuum was pulled blocking the leak path, having water in the lines caused some corrosion or other deposit which plugged the leak.

The helium leak at plugged position 1 right side measured before the static water test is the same, or slightly less than the leak rate when the target was first removed from the beam line.  There were no T1 vacuum issues during the running period, therefore no issues are expected when this target is put in service again.

 The following work was performed on the T1-MK1 target:

- Unused proximity sensor removed
- Lift flange removed, painted, relabeled
- Seals changed for feedthrough #1, #2, plug port, and both water supply feedthroughs
- Seals (2) changed for feedthorugh #3 which required desoldering protect monitor connector and machining o-ring support tube
- Replaced nylon ferrules for water supply swagelok fittings
- Leak checked upper water supply tubes: all OK

- Target rewired by David Cameron (motor wires OK, micro switch wires were re-terminated, but not replaced)

- Installed lift flange
- Checked potentiometer, motor, micro switches, profile monitor limit switches: all OK
- Checked profile monitor actuation: smooth motion, travel starts at 10psi, fully actuated at 35psi
- Leak checked target ladder: results OK, see notes attached
- Noticed protect monitor wire conduit had dropped down from vacuum flange.  It is secure in current position and connector would have to be de-soldered to lift it, so decided to leave as-is.

- Target ladder moved to position zero
- Profile monitor raised to 'IN' position and secured for transport
- Target transported from the hot cell to the beam line

Note that after the target was installed in the beam line and vacuum was pumped down it required ~55psi to actuate the profile monitor.  Approximately 20psi more than when not under vacuum.

DRAWINGS UPLOADED IN PDF, DOWNLOAD TO VIEW ALL DRAWINGS, PREVI

---------------------------------------------

Jan 27:

T1-MK1 was moved to the storage pit in position #4. M9BB was moved to the hot cell for shutdown maintenance (o-ring replacement, air cylinder servicing, etc.).

When M9BB work is done, it will be returned to the target station and replaced with T2-MK1 for servicing (target exchange, measurement target installation, etc.).

---------------------------------------------

Jan 29:

Maintenance operations on the M9BB were completed on the hot cell.

• The air cylinders actuated and lowered smoothly, and at 42 psi as indicated in the instructions document.
• The felt wiper on the shaft seal was replaced.
• 3 o-rings were replaced on the bronze shaft seal bracket and lubricated with vacuum grease
• upper limit switch actuation was confirmed after reassembly
• BB returned to the adapter plate on the turntable and reoriented for pickup

---------------------------------------------

Feb 03:

M9BB returned to target station:

• o-ring groove on monolith cleaned,
• replacement o-ring with vacuum grease placed
• during return, we noticed pieces of 'scrap metal' on the bottom of the beamblocker, it was removed and when scanned registered a count that exceeded the radiation cave monitor's dial, this will be stored/disposed as active waste.
• M9 ASU still disconnected, could not 'push-down' M9BB with gas line connected, gas line left disconnected for now.
• T handle plate removed, limit switch rod returned

T2-MK1 moved from target station to hot cell roof, north hook used for flask, south hook used to adjust the position of the alignment frame:

• section of hot cell roof railing had to be removed for the flask to clear when handled by the north hook, there is also a similar functioning cutout on the wall by the storage pits
• peak fields are 1100 uSv/hr at the hot cell opening by the target.

T2-MK1 servicing started on hot cell roof:

• T2-MK1 water flush completed, ~1 min air purge cycles completed for all target positions, left at position 5 (plugged) for longer term air purging (~3 hours).
• A broken ceramic cap was found on the top of the target vacuum flange.
• 4 quarter circle segments from what looks like the old-style graphite targets were found on the base of the target ladder.
• The coupling between the potentiometer gearbox and the ladder drive screw has a rubber section, we will have to inspect it and decide whether to replace it with a metal coupling during this shutdown, if so, we should also replace the plastic ferrules and update the bulkhead plate to be more servicable.

---------------------------------------------

Feb 05:

T2 station was surveyed, peak fields were found by the blanked off T2 opening: 4mSv/hr. Immediately at the cover fields were around 16 mSv/hr.

No fallen pieces of suspected graphite target segments were found on the monolith, it may be a good idea to check inside the monolith hole with a remote viewing camera to make sure nothing is obstructing the locating feature on the base of the target ladder (to be included in the T2 return work permit).

---------------------------------------------

Feb 10:

The T2-MK1 target ladder underwent a water rinse and several days of air purging. 

5 psi compressed air was run through each target position (including position 5 which is plugged) for several hours each. Then, the ladder was fully dried using the leak detector and  cold trap on Feb 07.

The leak detector was calibrated with Vacuum Group's external source to 8.7e-8 mbar-L/s prior to operation. The leak detector and vacuum line up to the target service panel connections were checked to be leak tight prior to testing (0 leak rate and <1e-3 Torr pressure).

A single pump-down cycle was sufficient to get vacuum to establish within reasonable time:

• Total pump-down duration was ~55 minutes
• 60 seconds to go into 'fine vacuum' mode.
• 2 min 13 seconds to get to ~1e-3 Torr vacuum in ladder
• leak rate was initially ~1e-8 Torr-L/s and slowly dropped to 0.0e-10 Torr-L/s over the course of the pumpdown (48 minutes to bottom out)
• the ladder was left at the bottomed out leak rate for around 5 minutes prior to venting. 
• approximately a 1cm thick, 10cm diameger puck of ice formed on the side of the LN2 vessel of the cold trap.

Upon closer inspection with Isaac Earle, the following were observed for graphite pieces on the T2-MK1 target ladder base:

• The surface finish on the chips strongly suggest that they are graphite target materials
• Curiously the graphite chips are all of different thicknesses, to our understanding, the graphite material used in the old targets are of consistent thickness.
• The rounded faces of the chips had no clear sign/evidence of brazing when compared to sample pieces that can be found in the Hot Cell Lab office.
• No clear signs of beam spots can be found on the flat faces of the chips, near the corner edge.
• No clear signs of brazing on the targets were observed.

The above observations suggest that the chips did not originate from an operational target that has seen beam.

---------------------------------------------

Mar 21:

The T2-MK1 target ladder was leak tested with 3psi x 0.5s doses of He.

The baseline initial leak rate of the ladder was ~1e-8 Torr-L/s but reached around 1e-10 Torr-L/s when allowed to pump down over ~1 hour. Fine test was reached after 30s of pumping down (with cold trap), ladder pressure was stable at ~1e-3 Torr throughout.

See attached image for detected leak rates, overall all joints performed well, with the highest leak rate of 4.2e-9 Torr-L/s registered by one of the joints on ladder position 1.

---------------------------------------------

April 02:

The T2-MK2 target ladder was moved to the target station from storage pit #3.

During the return procedure, from visual inspection of the target tube, there doesn't seem to be any dropped material in base of the tube.

We encountered some difficulty seating the o-ring on the monolith flange's groove because the o-ring was stored in a coil. Extra vacuum grease was used to keep the o-ring stuck flush in the groove.

When lowering the ladder onto the monolith flange, the chain hoist bottomed out just short and we had to lower the flask with the crane for the target ladder to make contact. Edi from vacuum group helped us pump down on the vacuum volume and at first glance it seems to be pumping down ok.

We will return in the coming days to re-do the service connections to the target ladder and do a final check of the system.

---------------------------------------------

April 03:

The plug in pos#5 and the 5cm target (SER# 301) in pos#3 on the T2-MK1 target ladder were removed in the hot cell and bagged and moved to the East hot cell. 

The loose graphite material on the base of T2-MK1 was also bagged and labelled and moved to the East cell. 

The measurement target bracket TRH1766 was installed in position 5 and a 5cm target cassette (SER # 308) was installed in position 3.

The target bracket was cleaned in an ultrasonic cleaner with simple green, then tap water before being dried. The tube screws were switched with DIY vented screws.

The pos 3 and 5 male Swagelok threads and conical sealing face were 'cleaned' with the conical scotch-brite tools (cone and cup) with the air ratchet and blown out with a compressed air can.

The Swagelok nuts on the new target cassette and bracket was first tightened till the Swagelok gap inspection gauge could not fit in the gap.

The ladder was then leak checked and initially only pumped down to 3e-3 Torr with 2.5e-8 T-L/s leak rate. After some tightening of the Swagelok nuts on pos 3 and pos 5, the pressure and baseline leak rate eventually bottomed out to 1e-3 Torr and 0.0e-10 T-L/s respectively.

With 3psi x 0.5s He, the highest leak rates were found in position 2b, 3b, and 5a, all being less than 3e-9 Torr-L/s.

Of note, there is a 'ding' on the 'right' side bellows (as viewed from hot cell widow with target ladder rotated closer toward the window), the highest leak rate was found when we sprayed He directly at this 'ding'.

 ---------------------------------------------

April 03 (Continued):

We moved the T2 flask alignment frame off from the T2 station, re-connected all target services to T2-MK2 and re-connected the compressed air line to the T2/M9 blocker.

The T2-MK2 profile monitor drives in/out properly without timing out.

The target ladder drives up and down smoothly between pos 0-3.

We tuned the demin flowrate to 1.1-1.2 gpm for the T2 target in position 3 (5cm Be), and for the T1 target in position 1 (12mm Be, actually done end of Mar).

After some time allowing the pump to flow, all process variables in the cooling package reached operational ('green') levels.

As normal though, the T2 target flowrate was relatively low and close to the lower warn limit of 3gpm. We should look into upgrading the pump on this cooling package or adjust the warn/trip limits according to some Engineering analysis.

 ---------------------------------------------

April 09:

• We attempted to open the needle valve on the T1 heat exchange further but it was already fully open. Flow through the heat exchange is still fluctuating around 6.3 GPM, occasionally dipping below 6 GPM. It may be possible that the heat exchange paddle wheel needs to be changed for the flow rate to increase again.

 ---------------------------------------------

April 11:

• After re-inspecing the supply and return lines for the T1 heat exchanger in the BL1A tunnels, no metering/flow adjusting valve was found (also true for the T2 line). The flow through the T1 heat exchange will have to hover around the 6gpm mark (flow changed likely because other upstream systems are receiving flow).
• The flow through the T2 target (12mm Be) hovers around 3GPM, for both position 3 and 5 so we will have to deal with it hovering around the warn limit for the coming operational period (pending thermal analysis to justify changes to the warn limit)
• The M9/T2 and M20 blocker movement was tested and confirmed.
• The profile monitor movement for both the T1 and T2 target ladders were confirmed.
• We also tested a method of powering off the cooling pump by disconnecting the PLC signal/control cable to the 480V power box on the mezzanine.After disconnecting the signal cable, the T2 pump successfully turns off and upon reconnecting, the control room successfully regains control over the pump.

 The T1-MK1 target was transferred from the storage pit to the beam line.  Services were hooked up and the cooling package was started (operating normally).

Target ladder documentation and elevation values for the new target were delivered to operations.

The recently removed T1-MK1 target ladder was flushed with city water for ~30s at all target positions. It was then purged with compressed air at ~5 psi for ~1 minute at all target positions. We then left the target ladder in pos5 to fully extend the bellows and allow it to dry overnight.

This ELOG will be updated as we progress with work done to dry the target ladder and eventually exchange the spent targets.

-----------------------------------

UPDATE (Feb 02, 2024): in the morning, the target ladder was moved to pos4, and air was allowed to run through the morning, in the afternoon the ladder was moved to pos2 and air was ran. At the end of the day, the ladder was moved to pos 0 and air was ran through over the weekend.

-----------------------------------

UPDATE (Feb 05, 2024): the following procedures were completed.

1. the air connections on the target ladder was replaced with the leak test/vacuum connections as per instructions.
2. LN2 was taken from the ISAC-I facility and used to perform two pump-down cycles with the Agilent He leak detector + cold trap.
3. First pumpdown cycle: 3min45sec to go from roughing to fine pumping, left for a total pumpdown time of1hr20mins, a relatively small amount of ice formed on the cold trap cylinder (see image).
4. Second pumpdown cycle: 2min30sec to go from roughing to fine pumping, 4min30sec to reach 1e-3 Torr, 5min to reach < 1e-3 Torr. 
5. Due to good pumpdown in the second cycle, we proceeded with leak testing: Baseling 1.0e-10 Torr-L/sec | ~1 sec He applications at 3.5 Psi directed at fittings/welds/bellows. Worse case measured leak was 0.9e-7 Torr-l/sec in pos 13 of fig 12 of Document 46600 (see taken image for detailed measurements at various locations).

We will prepare an update to the procedure to better log the leak rate measurements and update this ELOG/the procedure. Note that ice removed from second pumpdown was actually comparable to the first (see image, ice knocked off during cold trap removal), but total pumpdown duration (including leak testing) was ~4 hours for the second pumpdown. 

 UPDATE (Feb 13, 2024): see attached record with new template.

T1-MK1 electrical checks were completed on the hot cell. Specifically the profile and protect monitors were checked (assistance from Micheal Donohoe and Holden Jones from 'probes group'/accelerator systems).

Reference WP C2024-10-30-3.

"WORK INSTRUCTIONS / REFERENCE DOCUMENTS: Perform blip test of target protect module by attaching HV oscillator to protect HV bias. Check that the oscillator signal is readable from U,D, L, R and HALO signal cables.

For profile monitor, connect PM cable assembly to 0518 MWC module in portable NIM crate. Connect 0518 "START" to ch2 of an oscilloscope and "ANALOG" to ch1. Set scope to trigger on Ch2. Connect HV oscillator to profile HV bias. Observe signals on all channel devisions, note any channels not responding."

----------------------------------------

Test outcome (paraphrased from text discussion with Micheal Donohoe)

Profile monitor functioning properly from test, but right 'plate' on protect monitor yielded different results to other plates. May need to be re-checked, pending comparisons with results from previous tests (issues with this because "different oscillator was used"). We may have also not used couplings for cables that were correct for that particular protect monitor plate.

----------------------------------------

In images attached, red cable used for profile monitor (connect to the vertical connector at the rear), the dark green wire is used to ground the system when the profile monitor is being checked. The grey cable bundle (with 'box') is used to check the protect monitor, with vertical pins located by the main lifting flange support bars in the front left of the target (see bottom right of first image).

----------------------------------------

Micheal and Holden returned on the 31st with Thomas Manson, they performed direct testing on the leads of the protect monitor connector (directly hook up leads on protect monitor connector to oscilloscope/multi-meter). All direct checks went well, the waveform issue from yesterday's testing likely resulted from the testing cable having issues.

Electrical testing on T1-MK1 is now complete. Probes group will work to update their documentation and possibly develop a process change to fabricate new cables to make the testing happen fully on the hot cell roof and have it complete faster.

Note: it may be a good idea to perform preventative maintenance on the protect monitor @ T1-MK1 the next time it is on the hot cell. Probes group members noted that the protect monitor components are nearing their end-of-life and it would be good to replace old components to avoid the protect monitor failing in the target station.

See Cyclotron Fault 15787. The target ladder motor failed to move to the commanded position during a test. The T1 area was uncovered and the motor assembly was inspected by M. Dalla Valle, A. Newsome, and A. Kong. It was determined that the rubber coupler which connects the motor shaft to the drive assembly had degraded and snapped. This is similar to an incident which recently occurred on T2. The coupler was replaced with an all-metal version. The system was tested by the DCR operators. They ran the target ladder to each extreme (position 5 to position 0) multiple times. The target ladder was also moved to a few positions which had targets installed, and the cooling package was energized to run water through and confirm functionality. All feedback sensors for the target ladder assembly and cooling package were observed to behave normally, as confirmed by the DCR. RH group confirmed visually that there were no observable issues. The system is now considered operational.

It is recommended to change all drive system coupler parts on both T1 and T2 during the next shutdown to prevent this issue from happening in the future. These parts should be inspected during the annual inspection of the T1/T2 assemblies as well.

Yesterday (April 18th) the T1 and T2 cooling package PRV outlet lines were routed to the active drain in the 1A tunnel.  All items for the safety and standards compliance upgrade for T1 and T2 as specified in Document-68861 are now complete.

T1-MK1 was moved from the target station to the hot cell roof. 

Radiation surveys report 140 mSv/hr at 0.5m for the bare T1-MK1 target ladder.

On the hot cell roof, fields are 220uSv/h at 0.5m.

T1-MK2 was subsequently transported from the storage pit to the target station. 

After the transport operation, the 1A vacuum volume was then pumped down, and reached approximatley 1/10 of the vacuum level prior to venting.

Update: a few hours after the vacuum volume started pumping down, vacuum levels stabilized to the level observed prior to venting (see image) - confirming that the o-ring seal on the target is good and that the transfer operation was a success. 

------------------------------------------ 

During the transport operation we noticed that the chain drive on the target transfer flask was slipping considerably more when powered through the power bar on the camera stand. 

This issue was remedied by powering the transfer flask through a dedicated plug, and will be noted in procedures for future transport operations.

A fault was reported June 21, 2022 (Fault 15033) regarding T1 inlet pressure warnings... the pressure reading slowly dropped over time as seen in the attached strip tool screenshot (B1A:T1CS:PGIN). The pressure reading issue is attributed to a fault pressure transducer. The transducer was replaced on Oct. 11, 2022 my M. Dalla Valle. It is currently functioning as expected and the system is operational.

The T1 cooling system was refilled with water via the expansion tank until the tank level read approximately 38 cm.

The system pump was restarted. All sensors displayed acceptable nominal values. Demin flow was tuned to 1.0 gpm.

 No leaks or abnormalities observed - system running normally.

NOTE: the needle valve for demin flow control adjustment was behaving strangely - it needed to be tapped when loosening to increase flow in order for results to be observed. Suggest replacement next shutdown (2023) - a calendar reminder was set to do this.

 On Friday Aug 31 at approximately 3am T1CS:FGSEC dropped to 0gpm flow.  This was confirmed to be a real reading when T1CS temperatures continued to rise steadily.  Beam was turned off, and T1CS later turned off by operators (temps continued to rise due to heat from pump).  The cooling package was uncovered in the morning, and the T1CS:SVSEC solenoid was replaced by Doug Preddy and Keith Ng.  This solved the problem and the package was restarted and then covered.

This is the latest of several failures of these new solenoids at T1/T2 since installation in the 2016 winter shutdown.  It is suspected that the "enhanced electronics" in this model of the valves is vulnerable to damage from radiation.  ASCO does not carry 24V DC valves in their "General Service" line which does not have the enhanced electronics.  Valves from other suppliers are being investigated.  If a suitable 24V DC model cannot be found then we can revert to the 110V AC ASCO valves which we know are reliable.  This will require relays and wiring to be done by electricians.

On Wednesday August 15th the T1 cooling package tripped off, and could not be restarted.  On the EPICS screen the pump device went into an immediate time-out each time we attempted to turn it on.  An electrician was involved in the debugging, and issues were found with the 480V supply.  The breaker switch in the 480V supply panel was found to be damaged, and was replaced.  The relay in the contactor box for the pump motor thermal protection was also replaced.  These new parts did not solve the problem.  Resistance measurements between the phases of the pump motor revealed ~10ohms between phases A-B, but ~250ohms from B-C and A-C (measured from inside the contactor box).  The T2 pump and a spare replacement pump were also tested, and had ~10ohms between all phases.  It was therefore concluded that the pump motor, or possibly the wiring between the contactor and motor had failed.  The T1 cooling package was uncovered and drained by the end of the day.

On August 16th the old pump was disconnected and de-wired.  Damaged wire insulation on the pump motor was found, which likely caused a short, leading to failure of the pump motor, and cascading failures of the electrical system and possibly also the control PLC.  A spare pump (Chempump GB-3K-1S) was installed with custom cut gaskets.  The pump was re-wired by an electrician.  The cooling system was re-filled, and start-up was attempted, however there were remaining issues with the electrical system.  By the end of the day various testing by Controls Group and Electrical Group determined that the pump motor contactor also had failed and required replacing.

On August 17th the contactor and over-current protector were replaced by an electrician.  When the breaker was switched on the pump unexpectedly started immediately.  This was found to be caused by the control system drive signal module being faulty causing the output to be stuck on.  Failure of this module may have been caused by excessive current draw due to the damaged contactor.  The failed module was replaced, and the PLC and IOC were both restarted.  After this the system worked normally.  The pump was inspected by Maico Dalle Valle, no leaks were observed and operation seemed normal.  Shielding above the cooling package was replaced, and BL1A was restarted.  Cyclotron Fault #11527 was returned.

A new spare contactor and over-current protector have been ordered (Allied Electronics LC1D09BD and LRD12, Requisition #1037940).  These will be labeled and given to the Electrical Group.  A new spare T1 / T2 pump has also been ordered (Chempump GB-3K-1S, Requisition #1038041) which will be stored in the RH Meson Hall Hot Cell Lab Tool Port Boot Box Area.

A data sheet and quote for the replacement pump are attached.

 The T1 cooling package outlet water pressure sensor (B1A:T1CS:PGOUT) was replaced this morning with a new unit.  Over the last few months the sensor readout had been steadily decreasing (from around 20psi to 4psi) with no corresponding decrease in inlet pressure, and no change in the circuit flowrate (this is how these pressure sensors typically fail).

After installing the new sensor the readback in EPICS returned to the expected value.  Although there are no alarms or trips associated with this signal, it is still useful as a diagnostic tool.

On 23rd October we restarted the T1 and T2 water packages after the mini shutdown and after the site power outage. The rotary collimator was reporting no water flow and it was assumed the valve coil (B1A:T1:SVCOL) had failed again. T1 water package was uncovered the next day (24th) and we went to change the coil. Water flow was returned to the collimator and T1 water package was re-covered.

During the replacement, one of the nearby pressure sensor wires accidentally made contact with a nearby paddle wheel sensor shorting out the output connectors and making the flow read between 8 to 1999 GPM. The offending wire was observed and removed after going back down to have a closer look.

It is not believed the coil failed from the site power outage on the 19th. The water packages had been left off since the beginning of the mini shutdown (2nd of October?).

 Around 10pm on Sunday Oct 22 beam was tripped multiple times due to low flow for the T1 collimator cooling circuit (B1A:T1:FGCOL).  Temperature of the collimator (TC7COL, TC8COL) was monitored with reduced beam current and no increase was observed leading to the conclusion that flow was OK and the flow gauge was faulty.  Normal operation continued until around 1:30am on Oct 26 when the collimator thermocouples both rose quickly, causing the beam to trip when they reached 60C.  At this point we concluded that there was no longer flow in the cooling circuit.  1A was defined off and shielding was removed down to the T1 cooling package.  The problem was found to be the solenoid valve which would open when actuated, but close in less than a second.  The flow meter was checked, appeared fine, and the paddle wheel was changed anyways.  The solenoid part of the valve was replaced (valve body not changed), which solved the problem.  Flow returned to 0.7gpm and the collimator thermocouples returned to normal value.  It is suspected that the valve was fluttering open and closed causing the decreased flow observed before, and then the valve finally failed closed at 1:30 on the 26th.

On Saturday July 21st the T1 Collimator solenoid valve would not re-open after tripping closed due to a (planned) loss of vacuum.  Shielding was removed and the valve solenoid was replaced which fixed the problem.  The original valve body was left in place.

This is the second time the solenoid at this location failed (See E-Log #239), and the latest of several failures of these new 24VDC solenoids at the T1 and T2 cooling packages.  Clearly they are not reliable in this radiation environment, so I will investigate an alternative to be installed in the 2018 Fall Shutdown or 2019 Winter Shutdown.

Cyclotron E-Fault #11420 has been returned.