Mike Russell from Diagnostics Group has completed an electrical check on the T1 and T2 profile and protect monitors.  He reports that both profile monitors look fine.  The T1 protect monitor looks fine except for a weak or missing signal from the halo monitor.  The T2 protect monitor looks fine.

Note: The protect monitor on the T1-MK2 target was working without issue during the previous running period and was not removed from the beam line during the 2016 shutdown.  The cable for the T1 protect monitor was not disconnected during the 2016 shutdown.

The following work was completed during the 2020 shutdown period for BL1A (Jan-Aug 2020):

• T1 and T2 EPICS data checked for anomalies.  None found, all sensors behaving as expected
• T1 and T2 cooling package water drained to storage tank, sampled, and released
• All cuno filters changed (3 for T1, 2 for T2)
• All T1 and T2 proteus paddle wheels changed
• T1 and T2 hansen fitting connections inspected: all good, no leaks
• Both packages refilled, restarted, demin flow tuned to 1gpm, and checked for leaks: very small leak from 3/4" line going into heat exchanger on T1 target loop (not new, tightening doesn't help) 

• Profile monitor air hoses inspected: ok
• Air amplifiers inspected: cycle time ok, no leaks
• M9 beam blocker overhaul completed including new air cylinders
• M9 and M20 BB actuation checked
• T1 and T2 profile monitor actuation checked
• T1 and T2 ladder movement checked
• XTPAGE 7L updated to indicate which targets spent
• Target ladder status sheet updated
• ASCO valves for M9BB, M20BB, T1 prof mon, T2 prof mon all serviced (with rebuild kit)

August 25, 2020 update:

- The cap block was replaced over T2
- Actuation of the M9-T2 and M20 beam blockers was tested.  Both moved fine and limits displayed correctly

 With-beam commissioning tests were completed today by Tony Tateyama and Isaac Earle.  The test results are attached in PDF format.  These tests are described in Section 7 of the Commissioning Plan (Document-118467 Release #2).  These tests were not part of the official commissioning, but intended as an additional check of system functionality with beam on, and to confirm that the CCS current protection system performs as expected.

Official commissioning of the T1/T2 EPICS system was completed in April 2016, with the results released in Document-131503.

 The following tasks were performed:

• All electrical, water, and pneumatic connections to the T1 target was reconnected.
• 3x and 2x CUNO filters for T1 and T2 respectively were changed (O-ring for the housings were not changed).
• All hansen fittings on the front mounting plates of the T1 and T2 cooling packages were inspected.
• All proteus paddle wheels (Q1-5) for both T1 and T2 were replaced. 
• The main reservoir drain valves on both cooling packages were closed in preparation for re-filling.
• All other ball valves in the cooling packages were returned to open, also in preparation for re-filling.

Two buckets containing (primarily) active water from the CUNO filters were brought closer to the boot up area, to be moved to the active sink in the MH hot cell lab in the coming days (pending ELOG update). 

Spent shafts, paddle wheels, and o-rings from the proteus flow meters will be checked and appropriately tagged/disposed of, also in the coming days.

------------------------------------------------

Some observations for future shutdown work:

• The drain port at the bottom of the CUNO filter housings can be used to verify that the filter line has been successfully closed off.
• Prepare several lengths of flathead screws to service the proteus flowmeters.
• Prepare spare locknuts and screws for the proteus flowmeters in case they become stripped or a locknut is dropped.
• Stick to well-fitting gloves to make using screwdrivers, picking screws, etc. easier.
• Possibly a pipe or coupling could be used with the tool for the CUNO filters to make it easier to loosen the CUNO filter nut in hard to reach areas.
• If possible, it may be a good idea to replace the o-rings on the CUNO filter housings in future shutdown operations (dash # to be identified). 

 The following tasks were completed in preparation of re-filling the T1 and T2 cooling packages:

1. All valves were returned to the 'open' configuration (note, compressed air to collimators at T1 is to remain closed - rotating collimator no longer in use || also note, odd handle configuration at T1 panel when all valves are open - see attached).
2. Leaks from valves were checked, two leakly valves found at T2 (Swagelok SS-44S6 and SS-45S12), the stem of these valves were re-tightened and leaks were remedied (see picures).
3. Spent CUNO filters in buckets were dealt with: water drained to MH HC active sink, and filters organized into a single 'dry' bucket.

The cooling packages will be filled to 38 cm at the expansion tank tomorrow, and the pumps turned on to check for leaks at pressure, at which point this ELOG will be updated.

UPDATE:

The cooling packages were filled on Jan 16 to ~38 cm, after which both pumps were turned on and a few more leaky valves were identified at the T2 station (none at T1).

We remedied the leaks by tightening their stems and will continue to monitor both systems for a few days.

Besides the leak, we found that the TGT:OUT pressure gage was incorrectly reading 0 psi. We will be entering the 1A tunnels to drain the 1A holding tank soon, at which point we will look to remedy this issue (either a valve was left closed, or an air column is trapped at the gage port).

UPDATE (30 Jan, 2024): TGT:OUT pressure transducer at T1 was successfully replaced and is now reading the correct pressure (see image, ~20 psi is about the same as readings from 2021 when the sensor was working properly).

 In preparation for Edi to vent the BL1A vacuum volume, the T1 and T2 cooling packages were both shut off.  They will remain off until vacuum is restored in BL1A in a few weeks time.  This afternoon, water will be purged from the T1 and T2 targets using compressed air so that stagnant water is not sitting in the targets for an extended period of time.

The air amplifier was also shut off and will remain off for the duration of the shutdown.  The lower of the two amplifiers was operating during the last running period.

At 9:15am this morning BL1A tripped off due to loss of the watchdog signals for the T1/T2 control system.  Graham Waters investigated and the problem was found to be a corrupted initialization file.  The error was corrected, and BL1A was started by 11:30am.
 

The T1/T2 Control System "Assembly Tests"  (Document 118467 Release #1, Section 5) have been completed.  The expected result, as written in the document, occurred for every test.  The T2 Shield Thermocouple #3  (B1A:T2:TC3SHLD:RDTEMP) showed slightly erratic behaviour when viewing archived data.  It is uncertain at this point whether or not it will need to be bypassed in the B1A:T2:TGTRDY interlock.

The official commissioning tests (Section 6) are currently on-going.

 - The target control software was modified to reference the correct pot. look-up table for the target at T1 (T1-MK1)
 - The target position buttons on the T1 & T2 panels were not working properly and Graham Waters was contacted to investigate
 - Graham power cycled the microprocessor and the TG1 crate which seemed to make things worse (the control software would not even boot up)
 - Graham was able to diagnose the problem remotely, and found a loose cable which was causing it not to boot
 - The target position issue will not be looked into further until a longer maintenance period or the 2013 shutdown
 - Target position can be controlled manually using a portable control box
 - Using the control box, the T1 ladder was moved to position zero to allow use of the profile monitor;  The ladder was then moved back to position 3
 

Fabrication of the T1/T2 beryllium target testing jig (TBP1721) was completed today by Dan MacDonald.  With a target installed across the test ports, the inside volume pressurized with air to 103psi, and both ball valves closed, there was no noticeable drop in pressure over a 16 hour period.  This result was deemed satisfactory for use of the jig for hydrostatic testing of new targets.

T1 and T2 beryllium target failure history was investigated back to 2000 (as far as XSTRIP records were readily available).  For each target cassette the following were noted: length of time the target was subject to direct beam, length of time the target was on the ladder w/o direct beam, and method of failure.  The purpose of the study was to devise a target replacement strategy so that T1 and T2 beryllium targets are preemptively retired from service to reduce the occurrence of failure while running which causes loss of beam time, extra dose to personnel, and damage to equipment.

The following recommendations are the result of the study:
- T1 beryllium targets should be retired after 9.0 months of running time
- T2 beryllium targets should be retired after 12.0 months of running time
- Multiple targets should be installed on each target ladder (at least 2, possibly 3) so that targets may be switched during a running period if required and less target assembly moves are required

A PDF copy of the Excel spreadsheet is attached which includes target use history, failure history, summary of results, and more specific recommendations for each target assembly.  This study and the replacement plan recommendations were reviewed and endorsed by Grant Minor and Yuri Bylinski.  This spreadsheet should be reviewed and updated every shutdown period (including Fall mini-shutdowns), as well as anytime there is a target failure.

Today we re-lubricated the o-rings on the top air amplifier for T1/T2, and replaced the o-rings on the bottom air amplifier with new and also lubricated o-rings. 

We used Haskel lubricant #50866.

The bottom air amplifier spool assembly was noticably dirtier than the top (see attached). and we noticed some worn spots on the o-rings. This was not the case with the top air amplifier spool (it has not been used outside of testing since being fully serviced in 2012).

Replacing the o-rings did not result in any perceptible change to the cycle rate and sound of either air amplifier (~50 sec/cycle, same sounds made by the piston before and after servicing). 

We initially planned to also remove the sleeve on the amplifiers (see image) but could not do so easily. We will contact Haskel to get some input regarding how best to remove the sleeve.

The plan is to perform a full teardown of the lower air amplifier next shutdown (2024).

The T1-P2 (outlet) pressure transducer was replaced to address an erroneous signal.  The XTPAGE value now reads as expected.

The T2-P1 (inlet) transducer was also replaced to address a noisy (and worsening) signal.  Replacing the transducer seems to have had no effect on the XTPAGE values.  A possible controls issue is suspected and Jamie Cessford will be notified so that the matter is directed to the appropriate person.

No dose was incurred during the procedure.

 The T1-MK2 target was wired for automatic target version detection by the new PLC control system (A to C jumper  in POT/CAM MS connector).  T1-MK1 and T2-MK1 targets still need to be wired which can take place after the shutdown without moving them from the storage pit.

After extensive testing over the last 2 weeks it was determined that the protect monitor on the T1-MK2 target had an issue with intermittent shorting between the high voltage line and ground (movement of the target ladder was causing the short to come and go).

On the May 21 maintenance day the T1-MK2 target was moved from the beamline to the hot cell (102mSv/hr @ 0.5m).  The T1-MK1 target was then moved from Storage Pit location #3 to the beam line (as in the past, alignment of the flask to the #3 position hole was challenging, and the flask had to be suspended above the alignment pins to achieve alignment with the hole - In the future this storage pit location should be avoided).

Cameras were mounted on the T1 alignment frame for remote viewing of alignment with the hole as well as flask door position and hook latch position.  The camera locations chosen provided useful views and did not interfere with the target or frame.  After procurement of additional mounting brackets the cameras will be permanently installed at these locations (already completed on the T2 frame).

The target moves were completed successfully, vacuum pump-down was normal, and electrical checks of the T1-MK1 profile monitor and protect monitor were normal.  When start-up of the cooling system was attempted problems were encountered with the control system.  Multiple channels on the digital output module were shorting, causing the power supply fuse in the PLC cabinet to blow.  This was eventually tracked down to two damaged cables (B1A:T1CS:SVSEC and B1A:T1:SVCOL) which somehow had been crushed during removal of shielding.  The cables were replaced and all cables secured as much as possible in the vertical chase leading up from T1.  After replacing the cables we discovered that the digital output for the SVSEC signal was stuck in 'drive' status.  The module was replaced and then the control system was working normally (approximately 11am on May 23rd).

The T1 cooling system was started (no change to 1ACG2 vacuum), ladder movement through all target positions was tested, and profile monitor actuation was tested.  Updated target info sheets were submitted to Ops (attached in PDF format) and XTPAGE 7L was updated.

Diagnosis and repair of the T1-MK2 target will take place in the hot cell over the coming weeks.

Cyclotron Fault #12266 was returned.

 The T1-MK2 target was transported from the storage pit to the beamline.  The target measured approximately 10mSv/hr at 0.5m.  All target services were hooked up and the cooling package was started (no change in T1 vacuum).

The following work was done to the T1-MK2 target in the hotcell between the dates of May 24, 2019 - Jan 30, 2020 (See E-Log #251 for further information):

• May 24: Flushed position 2 with water, purged all positions with air.  Left air running through position 0 over the weekend
• May 27: Switched to position 5, left air running to dry out bellows
• May 28: Started pump out with cold-trap;  Reached "fine test" (TP pressure < 10E-2 Torr) after ~10-15 minutes, left for 2 hours, removed ice build-up then restarted leak detector;  
  Down to 1.0E-3 Torr TP pressure and 1.0E-8 Torr*L/s after ~10min
  Leak checked with 5psi helium, ~0.5s spray: Response at lower delta seal right side (when facing ladder) up to 3.8E-6 Torr*L/s and drift responses from nearby locations
• June 27: Position 1 target removed - no issues, required light tapping with hamme
  Position 4 target removed - lots of hammer hits required
  Both target beam spots photographed.  Saw what looks like hole on Position 4 entrance window (see attached photos)
• July 8: Protect monitor inspected visually using Nikon level by Dave Cameron and Shengli Liu - Nothing abnormal observed
  Electrical check w/ multimeter: all OK
  Observed ground to HV plates while moving ladder from Position 5 to 0 - all OK
  Observed ground to HV plates while moving profile monitor IN then OUT - all OK
  Found intermittent short from HV to ground when shaking the target assembly or tapping the protect monitor cassette:  must replace
• July 18: Cleaned Position 1 and 4 Swagelok sealing surfaces on the target ladder using the Dremel tool with burgandy Scotchbrite
  Ready for new targets to be installed
  Plugs installed at Position 4
• July 24: Target #105 installed at Position 1
  Started leak detector pumpdown with cold trap;  Only pumped down to 4.0E-3 Torr;  Big helium response at lower right delta seal (10E-5 Torr*L/s range)
  Concluded that the leak got worse during the target replacement (likely from the hammering required to remove Position 4 target
  Tightened bolts for lower right delta seal then repeated leak check: same results, therefore must replace seals
  Received replacement protect monitor cassette from Probes Group and installed
• July 29: Probes group did blip test on new protect monitor - all OK
  We repeated the test for shorts with shaking and tapping on the target assembly and monitor - no shorts observed
  Concluded that protect monitor issues is now fixed
• Early Jan 2020: Lower cone and plate removed, ladder removed (old style w/o plate at top)
  Removed seal spacers, the one for the lower right delta seal was stuck, had to use a flat edge screwdriver as a wedge with aluminum hammer to dislodge
  Removed delta seals, observed some deposits or corrosion at lower right seal location
  Cleaned spacers using Scotchbrite on Dremel
  Used 3D printed jigs with glued on burgandy Scotchbrite to clean sealing surfaces (no air ratchet or dremel, polished with manipulators until a clean shiny surface was produced)
  Had to remove bolts for the ladder flanges to clean the surfaces, and when reinstalling one of them it cross threaded and couldnt be removed
  Bolted instead at different hole locations (still at opposite sides of flange) but also had to switch bolt locations on upper seals to do this because bolts will interfere if using same threaded hole
  Eventually got everything back together with new delta seals installed (TBP0454B, machine shop order #44138)
  Note: the upper delta seals were not disturbed - new bolts installed and torqued before old ones were removed
• Jan 22: Leak check, pumped with cold trap: down to 0.0E-10 Torr*L/s and 1.0E-3Torr TP pressure after 9.5 minutes
  Using 5 psi helium, 0.5s spray;  No response at any previous locations;  Very small drift response up to 0.5E-10 Torr*L/s
  Concluded that leak is fixed
• Jan 26: Base plate and cone reinstalled - no issues
• Jan 30: David Kyle and Shengli Liu re-tested the profile monitor and protect monitors - No issues
  Measured vertical and horizontal positions for all targets and protect monitor, updated measurements sheet (PDF attached)
  Ladder moved to Position 0
  Target assembly ready to be moved to storage pit
   

 The T1-MK2 target was moved from the hot cell to Hole #5 in the storage pit.  The M9 beam blocker was then moved from the beamline to the hot cell for MRO work.  A field of 14.4mSv/hr was measured at 0.5m from the beam blocker during removal.

During first injection of beam down 1A problems were encountered with the T1 protect monitor.  Testing was done by Probes Group from the mezzanine leading to the following investigation and repair work.

May 9th:

- T1 area uncovered, Probes Group performed testing at the top of the monolith confirming that the problem was below the vacuum flange
- T1-MK2 target moved from the beamline to the hot cell.  Measured field 800uSv/hr at ~0.5m
- Visual inspection of protect monitor performed:  No obvious issues with monitor plates.  Some small impact holes visible on exit side of monitor, similar to damage on the previous monitor from Oct 2015
- Electrical check performed on old protect monitor in hot cell, confirming the previous results
- Failed protect monitor removed, resistor pack installed and tested with expected results
- Profile monitor insulation found to be blackened and cracking, so the profile monitor cassette was also removed
- New protect monitor and profile monitors installed
- Electrical check performed on both monitors at the hot cell: Results good for both

May 10th:

- Vertical and horizontal positions measured for the new protect and profile monitors
- Profile monitor actuation tested:  started moving ~10psi, fully actuated ~50psi, monitor limit switches ok
- Measured total resistance of potentiometer: 20.26kOhms
- Target windows inspected and photos taken (stored on Isaac's hard drive), nothing unusual encountered
- T1-MK2 target returned to beamline, vacuum pumped down without issue
- T1 cooling package started fine with no change in vacuum
- T1 profile monitor actuated in and out.  Movement smooth and slow
- Electrical check done on protect and profile monitors from mezzanine: both good

The T1-MK2 target was moved from the hot cell to the storage area, in position 5. Note: manual nudging of the flask was required to get it to align with guide pins properly.