This is a continuation of work detailed in E-Log #129 and #140.

- Static water test performed on the target cooling lines:
     - Test setup assembled and leak checked by pressurising with house air, submerging in water and checking for bubbles (see photos)
     - System filled, pressurized, and valved off on Friday Feb 13 at 5:30pm,  Gauge read 60psi
     - Checked Monday Feb 16 @ 9:00am.  Gauge read 56psi
     - Checked Monday Feb 16 @ 5:00pm.  Gauge read 56psi
     - Checked Tuesday Feb 17 @ 9:30am.  Gauge read 55.5psi  (0.5psi / 24hrs rate)

- Checked results with Dimo and he requested a helium leak check be performed with a newer model leak detector to quantify leak rate:
     - Using Varian 979 leak detector, pumped ~1.5hrs w/ cold trap to remove water
     - TP press: 0.0*10^-4 torr,  Base leak rate ~7.0*10^-9 atm cc / sec
     - Tested all normal leak check locations and did general helium flood with no response at all  (1.5psi helium pressure)
     - Over ~ 20 minute period the base leak rate gradually rose to 8.8*10^-9 atm cc / sec, at one point spiked to high 10^-9 level, then returned to base line
     - Reviewed results with Dimo, he requested test be repeated after the target sits vented overnight

- Target vented and left overnight, leak check repeated Feb 18 starting 9am:
     - 0.0*10^-4 torr TP pressure after ~5 mins
     - 10^-9 atm cc / sec leak rate level after ~20 mins
     - All locations sprayed with 1.5psi helium, general flood in ladder area --> no response (Dimo and Edi present)
     - Target deemed OK for use in beam line by Dimo, he requested the cooling lines be filled with helium after installation in b/l while leak checking T1 volume

- Target will be transported to the storage pit tomorrow (Feb 19) and is ready for use as the spare T1 target

The target had an obvious helium leak when first tested (E-Log #129 and #140), which could not be found when helium leak checking again after the static water test.  Some possible explanations are:  water remaining in the lines blocked the leak path,  water remaining in the lines froze when vacuum was pulled blocking the leak path, having water in the lines caused some corrosion or other deposit which plugged the leak.

The helium leak at plugged position 1 right side measured before the static water test is the same, or slightly less than the leak rate when the target was first removed from the beam line.  There were no T1 vacuum issues during the running period, therefore no issues are expected when this target is put in service again.

 The following work was performed on the T1-MK1 target:

- Unused proximity sensor removed
- Lift flange removed, painted, relabeled
- Seals changed for feedthrough #1, #2, plug port, and both water supply feedthroughs
- Seals (2) changed for feedthorugh #3 which required desoldering protect monitor connector and machining o-ring support tube
- Replaced nylon ferrules for water supply swagelok fittings
- Leak checked upper water supply tubes: all OK

- Target rewired by David Cameron (motor wires OK, micro switch wires were re-terminated, but not replaced)

- Installed lift flange
- Checked potentiometer, motor, micro switches, profile monitor limit switches: all OK
- Checked profile monitor actuation: smooth motion, travel starts at 10psi, fully actuated at 35psi
- Leak checked target ladder: results OK, see notes attached
- Noticed protect monitor wire conduit had dropped down from vacuum flange.  It is secure in current position and connector would have to be de-soldered to lift it, so decided to leave as-is.

- Target ladder moved to position zero
- Profile monitor raised to 'IN' position and secured for transport
- Target transported from the hot cell to the beam line

Note that after the target was installed in the beam line and vacuum was pumped down it required ~55psi to actuate the profile monitor.  Approximately 20psi more than when not under vacuum.

 The T1-MK1 target was transferred from the storage pit to the beam line.  Services were hooked up and the cooling package was started (operating normally).

Target ladder documentation and elevation values for the new target were delivered to operations.

The recently removed T1-MK1 target ladder was flushed with city water for ~30s at all target positions. It was then purged with compressed air at ~5 psi for ~1 minute at all target positions. We then left the target ladder in pos5 to fully extend the bellows and allow it to dry overnight.

This ELOG will be updated as we progress with work done to dry the target ladder and eventually exchange the spent targets.

-----------------------------------

UPDATE (Feb 02, 2024): in the morning, the target ladder was moved to pos4, and air was allowed to run through the morning, in the afternoon the ladder was moved to pos2 and air was ran. At the end of the day, the ladder was moved to pos 0 and air was ran through over the weekend.

-----------------------------------

UPDATE (Feb 05, 2024): the following procedures were completed.

1. the air connections on the target ladder was replaced with the leak test/vacuum connections as per instructions.
2. LN2 was taken from the ISAC-I facility and used to perform two pump-down cycles with the Agilent He leak detector + cold trap.
3. First pumpdown cycle: 3min45sec to go from roughing to fine pumping, left for a total pumpdown time of1hr20mins, a relatively small amount of ice formed on the cold trap cylinder (see image).
4. Second pumpdown cycle: 2min30sec to go from roughing to fine pumping, 4min30sec to reach 1e-3 Torr, 5min to reach < 1e-3 Torr. 
5. Due to good pumpdown in the second cycle, we proceeded with leak testing: Baseling 1.0e-10 Torr-L/sec | ~1 sec He applications at 3.5 Psi directed at fittings/welds/bellows. Worse case measured leak was 0.9e-7 Torr-l/sec in pos 13 of fig 12 of Document 46600 (see taken image for detailed measurements at various locations).

We will prepare an update to the procedure to better log the leak rate measurements and update this ELOG/the procedure. Note that ice removed from second pumpdown was actually comparable to the first (see image, ice knocked off during cold trap removal), but total pumpdown duration (including leak testing) was ~4 hours for the second pumpdown. 

 UPDATE (Feb 13, 2024): see attached record with new template.

T1-MK1 electrical checks were completed on the hot cell. Specifically the profile and protect monitors were checked (assistance from Micheal Donohoe and Holden Jones from 'probes group'/accelerator systems).

Reference WP C2024-10-30-3.

"WORK INSTRUCTIONS / REFERENCE DOCUMENTS: Perform blip test of target protect module by attaching HV oscillator to protect HV bias. Check that the oscillator signal is readable from U,D, L, R and HALO signal cables.

For profile monitor, connect PM cable assembly to 0518 MWC module in portable NIM crate. Connect 0518 "START" to ch2 of an oscilloscope and "ANALOG" to ch1. Set scope to trigger on Ch2. Connect HV oscillator to profile HV bias. Observe signals on all channel devisions, note any channels not responding."

----------------------------------------

Test outcome (paraphrased from text discussion with Micheal Donohoe)

Profile monitor functioning properly from test, but right 'plate' on protect monitor yielded different results to other plates. May need to be re-checked, pending comparisons with results from previous tests (issues with this because "different oscillator was used"). We may have also not used couplings for cables that were correct for that particular protect monitor plate.

----------------------------------------

In images attached, red cable used for profile monitor (connect to the vertical connector at the rear), the dark green wire is used to ground the system when the profile monitor is being checked. The grey cable bundle (with 'box') is used to check the protect monitor, with vertical pins located by the main lifting flange support bars in the front left of the target (see bottom right of first image).

----------------------------------------

Micheal and Holden returned on the 31st with Thomas Manson, they performed direct testing on the leads of the protect monitor connector (directly hook up leads on protect monitor connector to oscilloscope/multi-meter). All direct checks went well, the waveform issue from yesterday's testing likely resulted from the testing cable having issues.

Electrical testing on T1-MK1 is now complete. Probes group will work to update their documentation and possibly develop a process change to fabricate new cables to make the testing happen fully on the hot cell roof and have it complete faster.

Note: it may be a good idea to perform preventative maintenance on the protect monitor @ T1-MK1 the next time it is on the hot cell. Probes group members noted that the protect monitor components are nearing their end-of-life and it would be good to replace old components to avoid the protect monitor failing in the target station.

See Cyclotron Fault 15787. The target ladder motor failed to move to the commanded position during a test. The T1 area was uncovered and the motor assembly was inspected by M. Dalla Valle, A. Newsome, and A. Kong. It was determined that the rubber coupler which connects the motor shaft to the drive assembly had degraded and snapped. This is similar to an incident which recently occurred on T2. The coupler was replaced with an all-metal version. The system was tested by the DCR operators. They ran the target ladder to each extreme (position 5 to position 0) multiple times. The target ladder was also moved to a few positions which had targets installed, and the cooling package was energized to run water through and confirm functionality. All feedback sensors for the target ladder assembly and cooling package were observed to behave normally, as confirmed by the DCR. RH group confirmed visually that there were no observable issues. The system is now considered operational.

It is recommended to change all drive system coupler parts on both T1 and T2 during the next shutdown to prevent this issue from happening in the future. These parts should be inspected during the annual inspection of the T1/T2 assemblies as well.

Yesterday (April 18th) the T1 and T2 cooling package PRV outlet lines were routed to the active drain in the 1A tunnel.  All items for the safety and standards compliance upgrade for T1 and T2 as specified in Document-68861 are now complete.

T1-MK1 was moved from the target station to the hot cell roof. 

Radiation surveys report 140 mSv/hr at 0.5m for the bare T1-MK1 target ladder.

On the hot cell roof, fields are 220uSv/h at 0.5m.

T1-MK2 was subsequently transported from the storage pit to the target station. 

After the transport operation, the 1A vacuum volume was then pumped down, and reached approximatley 1/10 of the vacuum level prior to venting.

Update: a few hours after the vacuum volume started pumping down, vacuum levels stabilized to the level observed prior to venting (see image) - confirming that the o-ring seal on the target is good and that the transfer operation was a success. 

------------------------------------------ 

During the transport operation we noticed that the chain drive on the target transfer flask was slipping considerably more when powered through the power bar on the camera stand. 

This issue was remedied by powering the transfer flask through a dedicated plug, and will be noted in procedures for future transport operations.

A fault was reported June 21, 2022 (Fault 15033) regarding T1 inlet pressure warnings... the pressure reading slowly dropped over time as seen in the attached strip tool screenshot (B1A:T1CS:PGIN). The pressure reading issue is attributed to a fault pressure transducer. The transducer was replaced on Oct. 11, 2022 my M. Dalla Valle. It is currently functioning as expected and the system is operational.

The T1 cooling system was refilled with water via the expansion tank until the tank level read approximately 38 cm.

The system pump was restarted. All sensors displayed acceptable nominal values. Demin flow was tuned to 1.0 gpm.

 No leaks or abnormalities observed - system running normally.

NOTE: the needle valve for demin flow control adjustment was behaving strangely - it needed to be tapped when loosening to increase flow in order for results to be observed. Suggest replacement next shutdown (2023) - a calendar reminder was set to do this.

 On Friday Aug 31 at approximately 3am T1CS:FGSEC dropped to 0gpm flow.  This was confirmed to be a real reading when T1CS temperatures continued to rise steadily.  Beam was turned off, and T1CS later turned off by operators (temps continued to rise due to heat from pump).  The cooling package was uncovered in the morning, and the T1CS:SVSEC solenoid was replaced by Doug Preddy and Keith Ng.  This solved the problem and the package was restarted and then covered.

This is the latest of several failures of these new solenoids at T1/T2 since installation in the 2016 winter shutdown.  It is suspected that the "enhanced electronics" in this model of the valves is vulnerable to damage from radiation.  ASCO does not carry 24V DC valves in their "General Service" line which does not have the enhanced electronics.  Valves from other suppliers are being investigated.  If a suitable 24V DC model cannot be found then we can revert to the 110V AC ASCO valves which we know are reliable.  This will require relays and wiring to be done by electricians.

On Wednesday August 15th the T1 cooling package tripped off, and could not be restarted.  On the EPICS screen the pump device went into an immediate time-out each time we attempted to turn it on.  An electrician was involved in the debugging, and issues were found with the 480V supply.  The breaker switch in the 480V supply panel was found to be damaged, and was replaced.  The relay in the contactor box for the pump motor thermal protection was also replaced.  These new parts did not solve the problem.  Resistance measurements between the phases of the pump motor revealed ~10ohms between phases A-B, but ~250ohms from B-C and A-C (measured from inside the contactor box).  The T2 pump and a spare replacement pump were also tested, and had ~10ohms between all phases.  It was therefore concluded that the pump motor, or possibly the wiring between the contactor and motor had failed.  The T1 cooling package was uncovered and drained by the end of the day.

On August 16th the old pump was disconnected and de-wired.  Damaged wire insulation on the pump motor was found, which likely caused a short, leading to failure of the pump motor, and cascading failures of the electrical system and possibly also the control PLC.  A spare pump (Chempump GB-3K-1S) was installed with custom cut gaskets.  The pump was re-wired by an electrician.  The cooling system was re-filled, and start-up was attempted, however there were remaining issues with the electrical system.  By the end of the day various testing by Controls Group and Electrical Group determined that the pump motor contactor also had failed and required replacing.

On August 17th the contactor and over-current protector were replaced by an electrician.  When the breaker was switched on the pump unexpectedly started immediately.  This was found to be caused by the control system drive signal module being faulty causing the output to be stuck on.  Failure of this module may have been caused by excessive current draw due to the damaged contactor.  The failed module was replaced, and the PLC and IOC were both restarted.  After this the system worked normally.  The pump was inspected by Maico Dalle Valle, no leaks were observed and operation seemed normal.  Shielding above the cooling package was replaced, and BL1A was restarted.  Cyclotron Fault #11527 was returned.

A new spare contactor and over-current protector have been ordered (Allied Electronics LC1D09BD and LRD12, Requisition #1037940).  These will be labeled and given to the Electrical Group.  A new spare T1 / T2 pump has also been ordered (Chempump GB-3K-1S, Requisition #1038041) which will be stored in the RH Meson Hall Hot Cell Lab Tool Port Boot Box Area.

A data sheet and quote for the replacement pump are attached.

 The T1 cooling package outlet water pressure sensor (B1A:T1CS:PGOUT) was replaced this morning with a new unit.  Over the last few months the sensor readout had been steadily decreasing (from around 20psi to 4psi) with no corresponding decrease in inlet pressure, and no change in the circuit flowrate (this is how these pressure sensors typically fail).

After installing the new sensor the readback in EPICS returned to the expected value.  Although there are no alarms or trips associated with this signal, it is still useful as a diagnostic tool.

On 23rd October we restarted the T1 and T2 water packages after the mini shutdown and after the site power outage. The rotary collimator was reporting no water flow and it was assumed the valve coil (B1A:T1:SVCOL) had failed again. T1 water package was uncovered the next day (24th) and we went to change the coil. Water flow was returned to the collimator and T1 water package was re-covered.

During the replacement, one of the nearby pressure sensor wires accidentally made contact with a nearby paddle wheel sensor shorting out the output connectors and making the flow read between 8 to 1999 GPM. The offending wire was observed and removed after going back down to have a closer look.

It is not believed the coil failed from the site power outage on the 19th. The water packages had been left off since the beginning of the mini shutdown (2nd of October?).

 Around 10pm on Sunday Oct 22 beam was tripped multiple times due to low flow for the T1 collimator cooling circuit (B1A:T1:FGCOL).  Temperature of the collimator (TC7COL, TC8COL) was monitored with reduced beam current and no increase was observed leading to the conclusion that flow was OK and the flow gauge was faulty.  Normal operation continued until around 1:30am on Oct 26 when the collimator thermocouples both rose quickly, causing the beam to trip when they reached 60C.  At this point we concluded that there was no longer flow in the cooling circuit.  1A was defined off and shielding was removed down to the T1 cooling package.  The problem was found to be the solenoid valve which would open when actuated, but close in less than a second.  The flow meter was checked, appeared fine, and the paddle wheel was changed anyways.  The solenoid part of the valve was replaced (valve body not changed), which solved the problem.  Flow returned to 0.7gpm and the collimator thermocouples returned to normal value.  It is suspected that the valve was fluttering open and closed causing the decreased flow observed before, and then the valve finally failed closed at 1:30 on the 26th.

On Saturday July 21st the T1 Collimator solenoid valve would not re-open after tripping closed due to a (planned) loss of vacuum.  Shielding was removed and the valve solenoid was replaced which fixed the problem.  The original valve body was left in place.

This is the second time the solenoid at this location failed (See E-Log #239), and the latest of several failures of these new 24VDC solenoids at the T1 and T2 cooling packages.  Clearly they are not reliable in this radiation environment, so I will investigate an alternative to be installed in the 2018 Fall Shutdown or 2019 Winter Shutdown.

Cyclotron E-Fault #11420 has been returned.

Flow stopped through the T1 collimator cooling loop around the same time as a power bump this afternoon.  The problem was narrowed down to failed solenoid valve B1A:T1:SVCOL by comparing coil resistance to other units without issue (~1.0ohm vs 3.6ohm for working units), measured at the T1/T2 PLC cabinet.  Shielding above the T1 package was removed and the solenoid coil was replaced with a new unit.  After replacement the valve works as expected and B1A:T1:FGCOL reading is normal.  I inspected the T1 package while it was uncovered: no leaks observed.

The following work was completed on both the T1 and T2 target alignment frames

• One of the short taper pins on the top of the frame was swapped for a longer version to simplify flask alignment to the frame (now one pin can be aligned at a time instead of both concurrently)
• Three permanently mounted cameras installed to view east/west alignment, north/south alignment, as well as hook, latch, and door position
• LED strip lights installed on the bottom of the frame for better illumination during alignment of target devices
• All video and power signals routed through a single connector (9-pin CPC) for ease of use
• Frame painted with blue Rustoleum metal paint
• Hole positions, alignment orientation, camera position, etc labeled on frame

The T1 and T2 cooling packages were inspected today while running.  No water leaks were observed at either package including the high-active loops and copper active. 

The T1 pump was fairly noisy (video was taken to record the sound).  This is the new pump installed in August 2018 (see E-Log #242).  Maico reports that it had a similar noise when started after installation in August, and they were unsure if it was a normal sound or not.  The pump it replaced ran quietly, as well as the pump currently running at T2.  Further investigation will be performed this shutdown to try and determine the cause of the noise and appropriate action.

Both packages have been turned off and pump contactor control signals disconnected in preparation for MRO work and power outage this weekend.

 The following work was performed on the T1 and T2 cooling packages during the 2018 shutdown:

- T1 and T2 cooling packages both sampled and drained to South TNF holding tank
- Water released after testing and approval from RPG

- T1 high active and M15 filters changed (demin. filter to be changed next year)
- Replaced all T1 Proteus paddle wheels, shafts, and o-rings
- Replaced T1 male and female #6 SS Hansen fittings at the package/hose interface
- Replaced T1 #4 female Hansen fittings for the resin can (now all Hansen fittings on the high active loop have been replaced)
- T1 package filled and started, no change to T1 area vacuum
- Demin flow tuned to ~1.0gpm
- Package inspected while running, no leaks observed

- T2 high active filter changed (demin. filter to be changed next year)
- Replaced all T2 Proteus paddle wheels, shafts, and o-rings
- Replaced T2 male and female #6 SS Hansen fittings at the package/hose interface
- Replaced T2 #4 female Hansen fittings for the resin can (now all Hansen fittings on the high active loop have been replaced)
- T2 resin can changed;  old resin can flushed with water then air in Warm Cell and left to decay
- T2 package filled and started, no change to T2 area vacuum
- Demin flow tuned to ~1.0gpm
- Package inspected while running, no leaks observed

The cooling packages are functioning normally and are ready for the running period.