This Monday morning, Chad and I leak checked heat shielding circuit on TM3 with new target installed. The base LR on LD :0.0xE-9 atm.cc/sec, base pressure 0.0xE-4 torr. Chad sprayed helium twice on the VCR joints. No leak has been found. The circuit will be pressured with helium at CS.

Chad and i did a leak check on TM3 heat shield and window circuits.Both circuits are pumped easily down to the bottom leak rate and base pressure on the leak detector. There are two VCR caps on heat shield circuit. Chad sprayed helium on both of them   ( tested individually,2 seconds every time ) . There is no response on leakdetecor during 45 seconds waiting time. On window circuit there are 2 VCR joints. Chad sprayed helim on these joints(2seconds for each), and no response on leak detector. All joints  are double checked(sprayed twice on each of them). The result is they are leak tight.

TM3 has been moved from silo to NHC. The move is successful. NHC ram stand and NHC protection railings are checked and installed. Everything works fine.

2021-03-05. TM3 has been moved from NHC to silo. The move is successful.

Frank and I removed Tm3 from TCS to SHC. The move is successful.

Bill and I moved TM3 from SHC to NW silo with a dummy febiad target installed on it. The move is smooth.

TM3 has been moved from ITW to S-W silo. The move is smooth.

I started vacuum system.

I started TCS water system. The chiller does not work properly still. But, It will not affect the TM3 HV conditioning.  HV cover is installed on TM3 in TH. The module is ready for HV conditioning.

TM3 has been moved  from SHC to TCS. The move is smooth.

TM3 has been moved  from SHC to TCS by David and Frank. The move is smooth.

TM3 has been moved  from SHC to TCS.The move is smooth.

I stopped the TM3 and TCS vacuum pumps and water system. The TM3 is disconnected and vented. It will be moved to silo this afternoon.

TM3 has been disconnected at CS. It is vented and ready to move.

TM3 has been connected in ITW. See attachment for checklist.

TM3 has been connected in CS. All water supplies are bypassed theTM3 on the top of the service cap. TM3 TP2 was difficult to start. It is started finally after couple of time try.The pump runs fine now. 

With TiC # 8 installed, TM3 has been moved from SHc to ITW. The move is smooth.

A complete TM3 electrical check at TCS under good vacuum has been done. Pin 14 to 60KV only has 8K ohm on ohmmeter check , and it's about 2Kohm @250V Megga. It's low in resistivity. The pin 14 is for thermocouples only so no big concern.  The rest things are good. See attachments.

TM3 has been disconnected this morning and  purged all HV water lines. I found the power plug for all turbo pump fans wasnt connected  during the last run. That maybe the one of the reason on that ITE TP3 failure.

Today the voltage was raised from 65kV to 65.7kV. After about 10min at this level TM3 started current limiting. TM3 was not sparking when this happened.

ISAC OPS turned it OFF and then tried to ramp it back ON, but could not do it due to the current limiting.

I turned OFF/ON the PS (locally) 20ming ago and and then tried to get it back from the Control Room, but around 4-5kV it started current limiting.

Turned everything OFF on EPICS for the time being. We'll investigate on Monday.

Over the night the sparking rate at 65kV was from 6 to 2 sparks per hour. It tripped OFF once over the night: backing valves closed, tripping the turbos, tripping IG1 and turning OFF the BIAS. At that time ISAC OPS got it back smoothly.

the containment box has been removed from TM3.

Initial Summary:

TM3 sparking behaviour in ITE has been very disruptive both with and without protons. Protons are at 60uA, target heater at 540A. Max voltage achieved so far is 51kV, limited by time and a current limit on the BIAS supply.

The ITE:EL trips off every couple of minutes. Just needs to be restarted. Mike investigated and found that the trip is happening inside the power supply, then a moment later the canbus is defining the supply off when it cant get it to respond. Trip frequency decreased over time but still disruptive.

All the waterflow signals, including ones controlled in the cyclotron system, were tripping off regularly. These have all been forced, then we implemented a 5s delay in the ISAC waterflow interlocks. This seems to have solved the issue.

The ITE:BIAS supply has also been limited at 500uA instead of 5mA. Probably something hardcoded into the canbus - this was fixed by Keiko and the hardcode limit was set to 20%(?) of 5mA.

TM3 has been very sparky, even at low voltages. I can't understand why. A new ground path would be explicable but the excessive sparking is difficult to understand.

Possible avenues:

1. make sure the module outer surface is connected through a copper sheet to the sheet on the wall.
2. check that special ground still grounds all the gnd power supplies in the electrical room - this I checked but some of the grounds are connected through wires instead of sheets, this can be improved.
3. try running the module with only HV hooked up and nothing else.
4. try running TM3 in ITW.
5. Search work permits and shutdown jobs for any changes made to the system that could affect this.
6. hope there is a software solution for the EL problem, or start moving towards PLC control - no software solution because the canbus card is only reacting to the power supply refusing to turn on, so it seems the issue originates in the PSU. Moving to PLC is possible but may not help and increases the risk of frying the PLC module.
7. Add a spark gap or limiting resistor to the EL somewhere. Resistor only helpful if the EL itself is sparking. Difficult to find a spot for the spark gap.
8. Get a bipolar supply that can sink the current of a spark and replace the EL supply. - Tomislav doesn't seem to think this is viable.
9. Add protection on the controls cables.

Spark recording:

60uA/50kV/540A July 3 02:30 - 05:30 : average of 50 sparks per hour. data file https://isacwserv.triumf.ca/onlylocal/isacdata/ITE_TM3_SiC46_HVConditioning_2_20240704

60uA/32kV/540A July 3 19:15 - July 3 21:45 : average of 8.4 sparks per hour. data file https://isacwserv.triumf.ca/onlylocal/isacdata/ITE_TM3_SiC46_HPSIS_HVConditioning_20240704

50uA/42.8kV/540A July 4 8pm - July 5 2pm : average of 20 sparks per hour, or if you cut out the last two hour that were very sparky, 14 sparks per hour. data file http://isacwserv.triumf.ca/onlylocal/isacdata/TM3commissioning_July4_52024.dat

50uA/42.8kV/540A July 8 00:00 - July 8 18:00 : average of 3.4 sparks per hour. data file http://isacwserv.triumf.ca/onlylocal/isacdata/TM3commissioning_July82024.dat

50uA/42.8kV/540A July 10 10am - 3pm : 3.6 sparks per hour

50uA/53kV/540A July 17 5am - 10am : 4 sparks per hour

45uA/53kV/540A July 20 8pm - July 21 6:30am : 24 sparks in 11 hrs, avg 2.2 sparks per hour.

EL trip recording (first 2 entries from the archiver, verify with ops elog):

50uA.42.8kV/540A July 4 23:00 - July 5 16:00 : 9 trips, average 0.5 trips per hour

50uA/42.8kV/540A July 6 12:00 - July 7 03:00 : 6 trips with 2.5hrs off, average 0.5 trips per hour

50uA/42.8kV/540A July 7 05:00 - July 8 18:00 : no trips in past 30hrs. Last trip July 7 at 3am.

50uA/42.8kV/540A July 10 3pm - still no EL trips

50uA/53kV/540A July 16 8pm - July 17 10am : 3 trips in 6 hrs. Note cable in EL circuit got burned so EL is running at only ~4kV

45uA/53kV/540A July 17 3pm - July 18 9:30am : 7 trips in 18hrs

45uA/53kV/540A July20 14:00 - July 21 6:30 : 5 trips in ~17hrs

2024-11-15 EDIT, Alexander Shkuratoff:

I was not involved in this run, but I have gone through all the Elogs for TM3 up to this point and summarized all the data dumps at the bottom, so I may as well do it here.

	ITE_TM3_SiC46_HPSIS_HVConditioning_20240704	2024-07-04 01:42	4.8M
	ITE_TM3_SiC46_HVConditioning_Vac_20240704	2024-07-04 01:59	5.3M
	ITE_TM3_SiC46_HVConditioning_2_20240704	2024-07-04 05:31	1.1M
	TM3commissioning_July32024.dat	2024-07-04 17:04	11M
	TM3commissioning_July4_52024.dat	2024-07-05 14:18	12M
	TM3commissioning_July82024.dat	2024-07-08 18:03	13M

 Additionally, there is no Elog entry for August 18, 2024 for which TM3 commissioning continued in ITW, but I have found the following StripTool data:

	TM3commissioning_Aug182024.dat	2024-08-18 05:37	14M
	TM3commissioning_Aug182024_2.dat	2024-08-19 06:16	14M