Logbooks Lab Maintenance Evaporator_1 Evaporator_2 Laser cutter Target Production Test-Stand RH-ISAC RH-Cyclotron RH-Meson Hall RH-Beamlines RH-ARIEL
  RH-ISAC, Page 54 of 141  Not logged in ELOG logo
New entries since:Wed Dec 31 16:00:00 1969
ID Date Author Category Type Moduledown Target/Number Subject
  2622   Monday, June 24, 2024, 14:37 Carla BabcockSouth Hot-CellDevelopmentTM3UCx#44Inspection of TM3 and UCx#44 after no-protons run

TM3 and UCx#44 ran in ITE but issues with the cyclotron prevented proton beam delivery. Nonetheless, the target was heated and provided stable beam to TITAN as well as for tuning and some tests.

Results of inspection of TM3 module after run:

The module looked pretty good. There were no marks on the polished surface of the containment box, next to the water blocks, that were visible. But we didn't remove the box, only looked with the camera and it was visible by turning the module. So some things may have been hidden. There were some black marks on the door, near the bolts that hold in the handle (see attachment). This seems a very unlikely spot for sparking as the fields should be considerably lower than at the target handles, lower down. Maybe the spots are from oil left on the door that had a chemical reaction? There is one very faint matching spot on the top of the copper trident connector, but it doesn't look much like a spark mark. The other two marks have no corresponding mark on the source tray. There were no marks on the ground electrode, or anywhere else on the source tray. There was a scuff on one of the trapezoidal insulators, but it could have happened during installation.

 

Results of inspection of the target after run:

The FEBIAD target developed the same cathode short we have seen on almost every FEBIAD we run online - a high resistance short from cathode to HVC that prevents voltage on the cathode. This time the short remained after cooling the target, and David measured 208kOhms with the ohmmeter on the module top (11kOhms with the megger). We used a multimeter in the hot cell to test after the target had been dismounted and we found no short. So it must be related to either the position of the target (upright vs face down) or time (a conductive trace on an insulator oxidizes and becomes non-conductive again) or some pressure/shaking during the de-installation.

There weren't many marks on the target - a faint mark near where the faint mark on the copper trident was observed, some marks on the EE which had no counterparts on the ground electrode. See pictures.

Attachment 1: MicrosoftTeams-image_(61).png
MicrosoftTeams-image_(61).png
Attachment 2: MicrosoftTeams-image_(62).png
MicrosoftTeams-image_(62).png
Attachment 3: MicrosoftTeams-image_(60).png
MicrosoftTeams-image_(60).png
Attachment 4: MicrosoftTeams-image_(63).png
MicrosoftTeams-image_(63).png
  2623   Tuesday, June 25, 2024, 11:59 David WangITEStandard OperationTM3SiC #46Tm3 has been moved from TCS to ITE.

Tm3 has been moved from TCS to ITE. the move is smooth.

  2624   Tuesday, June 25, 2024, 11:59 David WangITEStandard OperationTM3SiC #46Tm3 has been moved from TCS to ITE.

Tm3 has been moved from TCS to ITE. the move is smooth.

  2625   Tuesday, June 25, 2024, 12:49 Frank SongITEStandard OperationTM3SiC#46Module Connection

 TM3 with new target SiC#46 was connected in ITE.

Attachment 1: TM3_with_SiC#46_connection_in_ITE_2024-06-25.pdf
TM3_with_SiC#46_connection_in_ITE_2024-06-25.pdf TM3_with_SiC#46_connection_in_ITE_2024-06-25.pdf TM3_with_SiC#46_connection_in_ITE_2024-06-25.pdf TM3_with_SiC#46_connection_in_ITE_2024-06-25.pdf
  2630   Thursday, July 04, 2024, 01:20 Carla BabcockITEDevelopmentTM3SiC#46TM3 conditioning with protons

Initial Summary:

TM3 sparking behaviour in ITE has been very disruptive both with and without protons. Protons are at 60uA, target heater at 540A. Max voltage achieved so far is 51kV, limited by time and a current limit on the BIAS supply.

The ITE:EL trips off every couple of minutes. Just needs to be restarted. Mike investigated and found that the trip is happening inside the power supply, then a moment later the canbus is defining the supply off when it cant get it to respond. Trip frequency decreased over time but still disruptive.

All the waterflow signals, including ones controlled in the cyclotron system, were tripping off regularly. These have all been forced, then we implemented a 5s delay in the ISAC waterflow interlocks. This seems to have solved the issue.

The ITE:BIAS supply has also been limited at 500uA instead of 5mA. Probably something hardcoded into the canbus - this was fixed by Keiko and the hardcode limit was set to 20%(?) of 5mA.

 

TM3 has been very sparky, even at low voltages. I can't understand why. A new ground path would be explicable but the excessive sparking is difficult to understand.

Possible avenues:

  1. make sure the module outer surface is connected through a copper sheet to the sheet on the wall.
  2. check that special ground still grounds all the gnd power supplies in the electrical room - this I checked but some of the grounds are connected through wires instead of sheets, this can be improved.
  3. try running the module with only HV hooked up and nothing else.
  4. try running TM3 in ITW.
  5. Search work permits and shutdown jobs for any changes made to the system that could affect this.
  6. hope there is a software solution for the EL problem, or start moving towards PLC control - no software solution because the canbus card is only reacting to the power supply refusing to turn on, so it seems the issue originates in the PSU. Moving to PLC is possible but may not help and increases the risk of frying the PLC module.
  7. Add a spark gap or limiting resistor to the EL somewhere. Resistor only helpful if the EL itself is sparking. Difficult to find a spot for the spark gap.
  8. Get a bipolar supply that can sink the current of a spark and replace the EL supply. - Tomislav doesn't seem to think this is viable.
  9. Add protection on the controls cables.

 

Spark recording:

60uA/50kV/540A July 3 02:30 - 05:30 : average of 50 sparks per hour. data file https://isacwserv.triumf.ca/onlylocal/isacdata/ITE_TM3_SiC46_HVConditioning_2_20240704

60uA/32kV/540A July 3 19:15 - July 3 21:45 : average of 8.4 sparks per hour. data file https://isacwserv.triumf.ca/onlylocal/isacdata/ITE_TM3_SiC46_HPSIS_HVConditioning_20240704

50uA/42.8kV/540A July 4 8pm - July 5 2pm : average of 20 sparks per hour, or if you cut out the last two hour that were very sparky, 14 sparks per hour. data file http://isacwserv.triumf.ca/onlylocal/isacdata/TM3commissioning_July4_52024.dat

50uA/42.8kV/540A July 8 00:00 - July 8 18:00 : average of 3.4 sparks per hour. data file http://isacwserv.triumf.ca/onlylocal/isacdata/TM3commissioning_July82024.dat

50uA/42.8kV/540A July 10 10am - 3pm : 3.6 sparks per hour

50uA/53kV/540A July 17 5am - 10am : 4 sparks per hour

45uA/53kV/540A July 20 8pm - July 21 6:30am : 24 sparks in 11 hrs, avg 2.2 sparks per hour.

 

 

EL trip recording (first 2 entries from the archiver, verify with ops elog):

50uA.42.8kV/540A July 4 23:00 - July 5 16:00 : 9 trips, average 0.5 trips per hour

50uA/42.8kV/540A July 6 12:00 - July 7 03:00 : 6 trips with 2.5hrs off, average 0.5 trips per hour

50uA/42.8kV/540A July 7 05:00 - July 8 18:00 : no trips in past 30hrs. Last trip July 7 at 3am.

50uA/42.8kV/540A July 10 3pm - still no EL trips

50uA/53kV/540A July 16 8pm - July 17 10am : 3 trips in 6 hrs. Note cable in EL circuit got burned so EL is running at only ~4kV

45uA/53kV/540A July 17 3pm - July 18 9:30am : 7 trips in 18hrs

45uA/53kV/540A July20 14:00 - July 21 6:30 : 5 trips in ~17hrs

 

2024-11-15 EDIT, Alexander Shkuratoff:

I was not involved in this run, but I have gone through all the Elogs for TM3 up to this point and summarized all the data dumps at the bottom, so I may as well do it here.

[   ] ITE_TM3_SiC46_HPSIS_HVConditioning_20240704 2024-07-04 01:42 4.8M
[   ] ITE_TM3_SiC46_HVConditioning_Vac_20240704 2024-07-04 01:59 5.3M
[   ] ITE_TM3_SiC46_HVConditioning_2_20240704 2024-07-04 05:31 1.1M
[   ] TM3commissioning_July32024.dat 2024-07-04 17:04 11M
[   ] TM3commissioning_July4_52024.dat 2024-07-05 14:18 12M
[   ] TM3commissioning_July82024.dat 2024-07-08 18:03 13M

 Additionally, there is no Elog entry for August 18, 2024 for which TM3 commissioning continued in ITW, but I have found the following StripTool data:

[   ] TM3commissioning_Aug182024.dat 2024-08-18 05:37 14M  
[   ] TM3commissioning_Aug182024_2.dat 2024-08-19 06:16 14M  

 

  2646   Tuesday, August 06, 2024, 09:16 Frank SongITEStandard OperationTM3SiC#46module disconnection

 TM3 with spent target SiC#46 was disconnected in ITE and ready to move to TCS soon.

Attachment 1: TM3_with_SiC#46_disconnection_in_ITE_2024-08-0106.pdf
TM3_with_SiC#46_disconnection_in_ITE_2024-08-0106.pdf TM3_with_SiC#46_disconnection_in_ITE_2024-08-0106.pdf TM3_with_SiC#46_disconnection_in_ITE_2024-08-0106.pdf
  2647   Thursday, August 08, 2024, 11:41 David WangSouth Hot-CellStandard OperationTM3SiC #46Tm3 was moved from ITE to TCS this Tuesaday, then to SHC this morning.

Tm3 was moved from ITE to TCS this Tuesday, then to SHC  this morning. Both moves are successful.

  2648   Thursday, August 08, 2024, 13:58 David WangSouth Hot-CellStandard OperationTM3SiC #46TM3 SiC#46 electrical check at SHC with target on (After operation)

ABCD is short to 60KV bias. Othjer are ok.  see attachment.

Found afterwards that this short was due to sparking through the multipin wire insulation where the wires pass behind the TBHT block "C". See fault 17265.

Attachment 1: TM3_Sic_46_after_run_E_check.pdf
TM3_Sic_46_after_run_E_check.pdf
  2650   Friday, August 09, 2024, 16:15 Frank SongSouth Hot-CellRepairTM3TiC#8electrical check/leak check

 TM3 with replaced target TiC#8 has been checked on SHC before moving to ITW. Pls see attachment

Attachment 1: electrical_and_leak_check_of_TM3_with_new_target_TiC#8_replaced_at_SHC_20240809.pdf
electrical_and_leak_check_of_TM3_with_new_target_TiC#8_replaced_at_SHC_20240809.pdf
  2652   Friday, August 09, 2024, 18:20 David WangITWStandard OperationTM3TiC#8TM3 has been moved from SHC to ITW.

With TiC # 8 installed, TM3 has been moved from SHc to ITW. The move is smooth.

  2653   Friday, August 09, 2024, 19:01 David WangITWStandard OperationTM3TiC#8.TM3 has been connected in ITW.

TM3 has been connected in ITW. See attachment for checklist.

Attachment 1: TM3_TiC#8_connection_in_ITW_checklist.pdf
TM3_TiC#8_connection_in_ITW_checklist.pdf TM3_TiC#8_connection_in_ITW_checklist.pdf TM3_TiC#8_connection_in_ITW_checklist.pdf TM3_TiC#8_connection_in_ITW_checklist.pdf
  2654   Sunday, August 11, 2024, 11:25 Carla BabcockSouth Hot-CellRepairTM3TiC#8short on multipin wires to TBHT discovered

With the new TiC#8 target installed on TM3, there was a ~7kOhm connection between TGHT/TBHT (A/B/C/D) and HV common. The standard electrical check on the module showed standard values, so it was assumed the problem was inside the target.

Started removing bolts/water connections one by one to see if any of them had an effect on the measurement. Found that disconnecting the multipin on the target from the mating connector on the source tray resolved the issue. Determined that the only way this could happen is if a spark cut through the insulation covering the multipin wires. Because of the way they are routed, they can touch the water lines and they are definitely touching waterline C, TBHT-. We do not typically check continuity of multipins to all water lines, just to HVC, so did not see this in the standard electrical check. Measurements verified that pin 2-4 was touching water line C, creating a short from TGHT/TBHT to HV common when the multipin connector is attached, since the pins are shorted to HVC inside the connector when they are not used.

Issue was solved by removing the target into the ante room, removing connector 1 all together since it was not used, and removing the grounding on the other pins, leaving only pin 2-2 connected for the anode.

In future we will change the IGLIS wiring so pin 2-4 is not used, and redesign the cable routing so it cannot touch the blocks. For the immediate term, we will add to the electrical check a check of each pin to TBHT/TGHT/COIL.

  2658   Tuesday, August 13, 2024, 14:23 David WangITEStandard OperationTM3Ta#67.TM2 with Ta#target has been moved from SHC to ITE.

TM2 with Ta#target has been moved from SHC to ITE. the move is smooth.

  2671   Tuesday, August 27, 2024, 13:30 Frank SongITWMaintenanceTM3TiC#8module disconnection

 TM3 is disconnected and all water lines are purged/refilled with 50psi Nitrogen. Pls see pics attached with.

Attachment 1: TM3_with_TiC#8_disconnection_in_ITW_2024-08-27.pdf
TM3_with_TiC#8_disconnection_in_ITW_2024-08-27.pdf TM3_with_TiC#8_disconnection_in_ITW_2024-08-27.pdf TM3_with_TiC#8_disconnection_in_ITW_2024-08-27.pdf
  2672   Tuesday, September 03, 2024, 11:40 David WangITWStandard OperationTM3TiC#8TM3 has been moved from ITW to S-W silo.

TM3 has been moved from ITW to S-W silo. The move is smooth.

  2685   Wednesday, September 18, 2024, 13:40 Aaron TamSouth Hot-CellStandard OperationTM3TiC#8Target removal

 TiC#8 HP FEBIAD removed from TM3

Target sitting in SHC

1 Coil Conductor bolt fell under source tray 

Photo Dump uploaded

Attachment 1: Cover_Outside.jfif
Attachment 2: Cover_Inside.jfif
Attachment 3: Arc_Sign_(as_seen_before).jfif
Attachment 4: Arc_Sign_2_(as_seen_before).jfif
Attachment 5: View_High.jfif
Attachment 6: Bolts_Removed.jfif
Attachment 7: Source_Tray_no_target.jfif
Attachment 8: Taget_Bottom.jfif
Attachment 9: Target_Left.jfif
Attachment 10: Target_Top.jfif
Attachment 11: Target_right.jfif
Attachment 12: Camera_Angle_2.jfif
Attachment 13: Bolt_location_2.jfif
  2686   Wednesday, September 18, 2024, 13:47 David WangSouth Hot-CellStandard OperationTM3TiC#8Tm3 has been moved from silo to SHC.

Tm3 has been moved  from silo to SHC. The move is smooth.

  2687   Monday, September 23, 2024, 11:26 David WangSouth Hot-CellStandard OperationTM3 TM3 window line has been leak checked.

TM3 window line has been leak checked after containment box removal and installation for insulator installation on water cup C.   No leak.

  2688   Monday, September 23, 2024, 11:31 David WangSouth Hot-CellDevelopmentTM3 10 pins lectrical check result after insulator is installed on water cap C.

10 pins lectrical check result after insulator is installed on water cap C.

pin 14 to 60kv bias has 20 Kohm @250V Megga check. The rest check on ohmmeter and megga @250 are either OLor over 25Mohm which are good.

  2690   Monday, September 23, 2024, 16:06 Aaron TamSouth Hot-CellRepairTM3 Insulator Addition to Water block

 September 23, 2024 - C. Fisher 

  • Containment box removed
  • Insulator installed
  • New window line VCR seals installed
  • Dropped bolt from TiC#8 removed
  • Containment box reinstalled

September 23, 2024 - A. Tam

  • Containment box cover reinstalled 

Photos attached 

Attachment 1: c02c089c-d2f0-4ddb-9f06-3dd8b8d05afa.jfif
Attachment 2: 0fd6eb03-7d82-48af-8ef7-99fd481bf1aa.jfif
Attachment 3: 14247104-53dc-4b00-852d-3a860c4baeb8.jfif
ELOG V2.9.2-2455