The M9 service stand floor plate (TB23351) was installed in the M9 pit yesterday.  Prior to installation all 5 threaded inserts were tested using a 1/2"-13 stud tool on an RH pole.  The west most hole (shown in red below) had damaged threads and the stud could only be threaded in 2 turns.  All other locations (shown in green) had usable threads with 6-8 full turns possible with the stud tool.  The plate was installed using RH floor plate bolts (TB23382) at all four locations with viable threads.  The bolts were torqued to 30 ft*lbs which on a bench test was shown to fully compress the belleville washer.  The standard torque spec for 1/2"-13 stainless bolts (43 ft*lbs) was not used to allow easier disassembly in the future.

 The T1-MK2 target was moved from the hot cell to Hole #5 in the storage pit.  The M9 beam blocker was then moved from the beamline to the hot cell for MRO work.  A field of 14.4mSv/hr was measured at 0.5m from the beam blocker during removal.

 Annual MRO work on the T1 and T2 cooling packages is now complete.  The following tasks were performed:

• Water from both packages drained to the South TNF holding tank (samples taken)
• All Proteus paddle wheels, shafts, and o-rings replaced (5 each at T1 and T2)
• All Cuno filters changed
• Archived data for all sensor readbacks inspected.  Only faulty sensor was B1A:T1CS:FGDEM
• Both packages refilled and restarted; Operating smoothly; Inspected for leaks, none found; Demin flow at ~1.0gpm
• B1A:T1CS:FGDEM readback is fixed with new paddle wheel
• South TNF holding tank water released after approval from RPG
• Old filters added to plastic pail with other T1/T2 filters located directly north of the BL1A blocks boot-box area
• Active water from filter canisters dumped into the RH lab active sink

The following work was done to the T1-MK2 target in the hotcell between the dates of May 24, 2019 - Jan 30, 2020 (See E-Log #251 for further information):

• May 24: Flushed position 2 with water, purged all positions with air.  Left air running through position 0 over the weekend
• May 27: Switched to position 5, left air running to dry out bellows
• May 28: Started pump out with cold-trap;  Reached "fine test" (TP pressure < 10E-2 Torr) after ~10-15 minutes, left for 2 hours, removed ice build-up then restarted leak detector;  
  Down to 1.0E-3 Torr TP pressure and 1.0E-8 Torr*L/s after ~10min
  Leak checked with 5psi helium, ~0.5s spray: Response at lower delta seal right side (when facing ladder) up to 3.8E-6 Torr*L/s and drift responses from nearby locations
• June 27: Position 1 target removed - no issues, required light tapping with hamme
  Position 4 target removed - lots of hammer hits required
  Both target beam spots photographed.  Saw what looks like hole on Position 4 entrance window (see attached photos)
• July 8: Protect monitor inspected visually using Nikon level by Dave Cameron and Shengli Liu - Nothing abnormal observed
  Electrical check w/ multimeter: all OK
  Observed ground to HV plates while moving ladder from Position 5 to 0 - all OK
  Observed ground to HV plates while moving profile monitor IN then OUT - all OK
  Found intermittent short from HV to ground when shaking the target assembly or tapping the protect monitor cassette:  must replace
• July 18: Cleaned Position 1 and 4 Swagelok sealing surfaces on the target ladder using the Dremel tool with burgandy Scotchbrite
  Ready for new targets to be installed
  Plugs installed at Position 4
• July 24: Target #105 installed at Position 1
  Started leak detector pumpdown with cold trap;  Only pumped down to 4.0E-3 Torr;  Big helium response at lower right delta seal (10E-5 Torr*L/s range)
  Concluded that the leak got worse during the target replacement (likely from the hammering required to remove Position 4 target
  Tightened bolts for lower right delta seal then repeated leak check: same results, therefore must replace seals
  Received replacement protect monitor cassette from Probes Group and installed
• July 29: Probes group did blip test on new protect monitor - all OK
  We repeated the test for shorts with shaking and tapping on the target assembly and monitor - no shorts observed
  Concluded that protect monitor issues is now fixed
• Early Jan 2020: Lower cone and plate removed, ladder removed (old style w/o plate at top)
  Removed seal spacers, the one for the lower right delta seal was stuck, had to use a flat edge screwdriver as a wedge with aluminum hammer to dislodge
  Removed delta seals, observed some deposits or corrosion at lower right seal location
  Cleaned spacers using Scotchbrite on Dremel
  Used 3D printed jigs with glued on burgandy Scotchbrite to clean sealing surfaces (no air ratchet or dremel, polished with manipulators until a clean shiny surface was produced)
  Had to remove bolts for the ladder flanges to clean the surfaces, and when reinstalling one of them it cross threaded and couldnt be removed
  Bolted instead at different hole locations (still at opposite sides of flange) but also had to switch bolt locations on upper seals to do this because bolts will interfere if using same threaded hole
  Eventually got everything back together with new delta seals installed (TBP0454B, machine shop order #44138)
  Note: the upper delta seals were not disturbed - new bolts installed and torqued before old ones were removed
• Jan 22: Leak check, pumped with cold trap: down to 0.0E-10 Torr*L/s and 1.0E-3Torr TP pressure after 9.5 minutes
  Using 5 psi helium, 0.5s spray;  No response at any previous locations;  Very small drift response up to 0.5E-10 Torr*L/s
  Concluded that leak is fixed
• Jan 26: Base plate and cone reinstalled - no issues
• Jan 30: David Kyle and Shengli Liu re-tested the profile monitor and protect monitors - No issues
  Measured vertical and horizontal positions for all targets and protect monitor, updated measurements sheet (PDF attached)
  Ladder moved to Position 0
  Target assembly ready to be moved to storage pit
   

 The second (and final) 55 gallon drum containing active items from the Meson Hall East Hot Cell was transported today from the Warm Cell to the Cyclotron East Vault Tunnel.   The updated form is attached.

The following items were moved today from the Warm Cell to the Rad Waste Pit for long term storage:

1. A 55 gallon drum containing various active components from the East Hot Cell.  The drum was labelled on the body and lid with the identification number RWC-55-01-2019.

2. The old M9 Q1Q2 service stand, which was removed from the beamline in the 2019 winter shutdown

Photos taken during the move are attached, as well as a copy of the loading record form for the drum.  The form has also been given to RPG.

A second drum (RWC-55-02-2019) remains in the Warm Cell, and will later be moved to the cyclotron vault tunnel for storage.  It was resurveyed after the first drum was removed and deemed too low-level for storage in the pit (300uSv/hr at 0.5m)

The following work was completed on both the T1 and T2 target alignment frames

• One of the short taper pins on the top of the frame was swapped for a longer version to simplify flask alignment to the frame (now one pin can be aligned at a time instead of both concurrently)
• Three permanently mounted cameras installed to view east/west alignment, north/south alignment, as well as hook, latch, and door position
• LED strip lights installed on the bottom of the frame for better illumination during alignment of target devices
• All video and power signals routed through a single connector (9-pin CPC) for ease of use
• Frame painted with blue Rustoleum metal paint
• Hole positions, alignment orientation, camera position, etc labeled on frame

After extensive testing over the last 2 weeks it was determined that the protect monitor on the T1-MK2 target had an issue with intermittent shorting between the high voltage line and ground (movement of the target ladder was causing the short to come and go).

On the May 21 maintenance day the T1-MK2 target was moved from the beamline to the hot cell (102mSv/hr @ 0.5m).  The T1-MK1 target was then moved from Storage Pit location #3 to the beam line (as in the past, alignment of the flask to the #3 position hole was challenging, and the flask had to be suspended above the alignment pins to achieve alignment with the hole - In the future this storage pit location should be avoided).

Cameras were mounted on the T1 alignment frame for remote viewing of alignment with the hole as well as flask door position and hook latch position.  The camera locations chosen provided useful views and did not interfere with the target or frame.  After procurement of additional mounting brackets the cameras will be permanently installed at these locations (already completed on the T2 frame).

The target moves were completed successfully, vacuum pump-down was normal, and electrical checks of the T1-MK1 profile monitor and protect monitor were normal.  When start-up of the cooling system was attempted problems were encountered with the control system.  Multiple channels on the digital output module were shorting, causing the power supply fuse in the PLC cabinet to blow.  This was eventually tracked down to two damaged cables (B1A:T1CS:SVSEC and B1A:T1:SVCOL) which somehow had been crushed during removal of shielding.  The cables were replaced and all cables secured as much as possible in the vertical chase leading up from T1.  After replacing the cables we discovered that the digital output for the SVSEC signal was stuck in 'drive' status.  The module was replaced and then the control system was working normally (approximately 11am on May 23rd).

The T1 cooling system was started (no change to 1ACG2 vacuum), ladder movement through all target positions was tested, and profile monitor actuation was tested.  Updated target info sheets were submitted to Ops (attached in PDF format) and XTPAGE 7L was updated.

Diagnosis and repair of the T1-MK2 target will take place in the hot cell over the coming weeks.

Cyclotron Fault #12266 was returned.

 Completed work at m9 beam line:

- Removed existing service stand and associated cable and water services. Legacy b1 water services still need to be removed for future magnet installation. Old service stand is currently in warm cell waiting for disposal.

- M9 Q345 and vacuum roughing line were cleaned by CMMS group.

- M9 B1 is disconnected but remains in beam line ready for removal and refurbishment for 2020 shut down. Relevant ports have been sealed off.

- New Q1 Q2 baseplate was test fit and did not fit with guide post bolted in place, interferes with T2 port blank off plate.

- 3d lidar scans of magnet space were done.

- Leica tracker data was taken of the holes in the poured in place concrete pad.


Photos from shutdown are on remote handling group folder, under "\\trwindata\groups\remote handling\Photos\2019\2019 m9 project"

The following work was completed between Jan 16 - Feb 19, 2019:

• T1 and T2 cooling packages drained to South TNF holding tank, water sampled and tested, released after approval from RPG
• T1 resin can swapped
• All Cuno filters changed at T1 (3 total)
• T1 inlet (T1CSPGIN) and outlet (T1CSPGOUT) pressure gauges replaced (archived data on EPICS indicated both starting to fail)
• T2 target water Cuno filter replaced
• All T1 and T2 solenoid valves changed to new models without power management circuit which will hopefully solve the problem of frequent failures (SASP0670)
  Parker brass body valve 73218BN4UN00 with solenoid C222C2 solenoid used for T1CSSVSEC and T2CSSVSEC
  STC stainless steel body valve 2S160-1/2-2-D with solenoid 2W200C-2-D used for T1CSSVTGT, T1SVCOL, T2CSSVTGT, T2SVCOLA, T2SVCOLB
  Necessary changes to PLC BOP wiring made by Tony Tateyama
  All new valves confirmed operational and leak tight
  Cyclotron Fault #11955 returned
• T1 and T2 profile monitor air cyclinder flow control valves replaced with new models tuned to ~1/8 turn open.  Part is Rego F125B (see PO #3044305)
• T1-MK2 profile monitor air cylinders replaced (both were leaking through the top seal)
• T1 and T2 profile monitor actuation checked - both move in and out slowly and smoothly
• M20 BB actuation checked - movement is slow and smooth
• T1 and T2 ladder movement checked - both ok
• T1-MK2 male Hansen fittings replaced as well as female Hansens which connect to them (these are the last remaining Hansen fittings to be replaced at T1/T2)
• Both cooling packages filled and started, now running smoothly.  Inspected for leaks (none found).  No change to 1A vacuum when packages started
   

Note 1: The T1 package water pump is slightly noisy when running (it has been this way since the replacement pump was installed last year) - Will investigate further at a later date

Note 2: In the week of Feb 11-15 Vacuum Group pressurized the T1 target cooling circuit with 30psi helium to try to find the vacuum load on the T1 volume.  This was done twice with the beamline vented for a few hours in between in case water had frozen at a leak location. The conclusion after testing is that the T1 cooling water is not the source of the vacuum load at T1.  Helium was also sprayed all around the top of the T1 monolith (T1 target and M15 permanent magnet) - there was no response on the leak detector

My total dose for this work was 0.15mSv

2018 Cyclotron lid up problem. Station #9 was  found to  have the motor wires along the beam plain had damaged wire insulation. There were bare wires in the conduit resulting in the wires shorting out. The wires were repaired with a quick short replacement. Lid went up for in tank work. After lid down the problem was investigated. We checked the radiation levels   at all  of the stations to come up with a plan to determine witch of the stations had the highest levels. We found that stations 1,3,5,7,9, and 11 had the highest levels. Station 1 was inspected and all wires were replaced to the T elbow. Station 3 all wires were replace to the T elbow. Station 5 all wires were replaced  to the T elbow. Station 7 all wires were inspected and were in good condition and put back into place. Station 9 all wires including syncro were replaced all the way to the junction box. Station 11 motor wires were good, but the jog wires and the syncro were replaced to the pull box. All of the jog switches on stations 1,3,5,7,9,and 11 were falling apart when wires were inspected.  New switches were ordered and assembled. At the start of 2019 shutdown all jog switches were replaced on stations 1,3,5,7,9, and 11. After all the jack work was finished the jog switches were check and were working well.

The T1 and T2 cooling packages were inspected today while running.  No water leaks were observed at either package including the high-active loops and copper active. 

The T1 pump was fairly noisy (video was taken to record the sound).  This is the new pump installed in August 2018 (see E-Log #242).  Maico reports that it had a similar noise when started after installation in August, and they were unsure if it was a normal sound or not.  The pump it replaced ran quietly, as well as the pump currently running at T2.  Further investigation will be performed this shutdown to try and determine the cause of the noise and appropriate action.

Both packages have been turned off and pump contactor control signals disconnected in preparation for MRO work and power outage this weekend.

Randomly checked on the nuclear ventilation for the Meson Hall Hotcells and discovered they were not running again (both magnahelic gauges displayed zero). Attempted to restart the fan manually and the control box seems to not be working. Matt McClean has been contacted and they will look into the problem.

The nuclear ventilation is currently not running in the RHMH hot cells.

On 23rd October we restarted the T1 and T2 water packages after the mini shutdown and after the site power outage. The rotary collimator was reporting no water flow and it was assumed the valve coil (B1A:T1:SVCOL) had failed again. T1 water package was uncovered the next day (24th) and we went to change the coil. Water flow was returned to the collimator and T1 water package was re-covered.

During the replacement, one of the nearby pressure sensor wires accidentally made contact with a nearby paddle wheel sensor shorting out the output connectors and making the flow read between 8 to 1999 GPM. The offending wire was observed and removed after going back down to have a closer look.

It is not believed the coil failed from the site power outage on the 19th. The water packages had been left off since the beginning of the mini shutdown (2nd of October?).

 On Friday Aug 31 at approximately 3am T1CS:FGSEC dropped to 0gpm flow.  This was confirmed to be a real reading when T1CS temperatures continued to rise steadily.  Beam was turned off, and T1CS later turned off by operators (temps continued to rise due to heat from pump).  The cooling package was uncovered in the morning, and the T1CS:SVSEC solenoid was replaced by Doug Preddy and Keith Ng.  This solved the problem and the package was restarted and then covered.

This is the latest of several failures of these new solenoids at T1/T2 since installation in the 2016 winter shutdown.  It is suspected that the "enhanced electronics" in this model of the valves is vulnerable to damage from radiation.  ASCO does not carry 24V DC valves in their "General Service" line which does not have the enhanced electronics.  Valves from other suppliers are being investigated.  If a suitable 24V DC model cannot be found then we can revert to the 110V AC ASCO valves which we know are reliable.  This will require relays and wiring to be done by electricians.

 The water in the South TNF Holding Tank was released today under work permit #C2018-08-28-3.  The tank contained approximately 100L from the T1 cooling package (drained Aug 16, 2018) and 350L from the 500MeV system (drained April 23, 2018 by Steve Sapriken).

On Wednesday August 15th the T1 cooling package tripped off, and could not be restarted.  On the EPICS screen the pump device went into an immediate time-out each time we attempted to turn it on.  An electrician was involved in the debugging, and issues were found with the 480V supply.  The breaker switch in the 480V supply panel was found to be damaged, and was replaced.  The relay in the contactor box for the pump motor thermal protection was also replaced.  These new parts did not solve the problem.  Resistance measurements between the phases of the pump motor revealed ~10ohms between phases A-B, but ~250ohms from B-C and A-C (measured from inside the contactor box).  The T2 pump and a spare replacement pump were also tested, and had ~10ohms between all phases.  It was therefore concluded that the pump motor, or possibly the wiring between the contactor and motor had failed.  The T1 cooling package was uncovered and drained by the end of the day.

On August 16th the old pump was disconnected and de-wired.  Damaged wire insulation on the pump motor was found, which likely caused a short, leading to failure of the pump motor, and cascading failures of the electrical system and possibly also the control PLC.  A spare pump (Chempump GB-3K-1S) was installed with custom cut gaskets.  The pump was re-wired by an electrician.  The cooling system was re-filled, and start-up was attempted, however there were remaining issues with the electrical system.  By the end of the day various testing by Controls Group and Electrical Group determined that the pump motor contactor also had failed and required replacing.

On August 17th the contactor and over-current protector were replaced by an electrician.  When the breaker was switched on the pump unexpectedly started immediately.  This was found to be caused by the control system drive signal module being faulty causing the output to be stuck on.  Failure of this module may have been caused by excessive current draw due to the damaged contactor.  The failed module was replaced, and the PLC and IOC were both restarted.  After this the system worked normally.  The pump was inspected by Maico Dalle Valle, no leaks were observed and operation seemed normal.  Shielding above the cooling package was replaced, and BL1A was restarted.  Cyclotron Fault #11527 was returned.

A new spare contactor and over-current protector have been ordered (Allied Electronics LC1D09BD and LRD12, Requisition #1037940).  These will be labeled and given to the Electrical Group.  A new spare T1 / T2 pump has also been ordered (Chempump GB-3K-1S, Requisition #1038041) which will be stored in the RH Meson Hall Hot Cell Lab Tool Port Boot Box Area.

A data sheet and quote for the replacement pump are attached.

On Saturday July 21st the T1 Collimator solenoid valve would not re-open after tripping closed due to a (planned) loss of vacuum.  Shielding was removed and the valve solenoid was replaced which fixed the problem.  The original valve body was left in place.

This is the second time the solenoid at this location failed (See E-Log #239), and the latest of several failures of these new 24VDC solenoids at the T1 and T2 cooling packages.  Clearly they are not reliable in this radiation environment, so I will investigate an alternative to be installed in the 2018 Fall Shutdown or 2019 Winter Shutdown.

Cyclotron E-Fault #11420 has been returned.

 The T1 cooling package outlet water pressure sensor (B1A:T1CS:PGOUT) was replaced this morning with a new unit.  Over the last few months the sensor readout had been steadily decreasing (from around 20psi to 4psi) with no corresponding decrease in inlet pressure, and no change in the circuit flowrate (this is how these pressure sensors typically fail).

After installing the new sensor the readback in EPICS returned to the expected value.  Although there are no alarms or trips associated with this signal, it is still useful as a diagnostic tool.

Flow stopped through the T1 collimator cooling loop around the same time as a power bump this afternoon.  The problem was narrowed down to failed solenoid valve B1A:T1:SVCOL by comparing coil resistance to other units without issue (~1.0ohm vs 3.6ohm for working units), measured at the T1/T2 PLC cabinet.  Shielding above the T1 package was removed and the solenoid coil was replaced with a new unit.  After replacement the valve works as expected and B1A:T1:FGCOL reading is normal.  I inspected the T1 package while it was uncovered: no leaks observed.