ITE entrance module top and module flange o-ing to tank have been leak checked. no leak at 0.0xe-9 atm.cc/sec. things are double checked.

The move is smooth and successful.

Tm3 has been connected in ITE.  see attachment.

ITE water signal annual interlock check is done. Ed helped me on this check. Everything works normal.

Tuesday, May 14, 2024, 10:26 :

Ramping up bias/EE/EL on ITE - will see if the HV cage in the pit can hold 60kV or not.

At around 50kV the bias read current starts to get jumpy. Doesn't look like its an issue for the voltage, but definitely instability increases.

Occasional sparks seem to increase the EL current significantly...

David heard some sparks in the target hall. Looks like from heat shield feed through to the top of the cage. Several sparks heard in the TH also changed the einzel lens draw current significantly. These feethroughs are not far from each other so could be related.

Apparently the sparking also tripped off all the optics and heaters in ITW, by changing the state of the water flow interlock. Interesting and not seen ever before. Will test again with the cage off, rather than risk tripping ITW again.

Tried ramping up the voltage on just the EE. Seems the limit is 7kV - at that level it gets sparky and unstable. Should be investigated (should hold 10kV) but not critical for module function.

Wednesday, May 15, 2024, 10:05 :

HV cover was removed this morning and the MAA was locked out.

Started ramping things up, was very easy until around 58kV. At 59.5kV a spark tripped off the EL. Canbus controller had to be power cycled to fix it. Bias current is much more stable in this configuration.

Ramped up again while listening from the TH to see if it was sparking outside. Went slowly to 60kV (left EL at 29kV) and didn't see any outside sparks in the first 10 mins, couple sparks in vacuum. But after 15 mins a spark seems to have fried the EL canbus card. We were not in the TH so not sure if it was on the outside or not.

After 2 hours a spark tripped off both ITE and ITW. Looks like it makes a water interlock go bad momentarily, but after the trip the interlocks seem fine.

Tuesday, May 28, 2024, 16:41

Running TM3 at any voltage in ITE produces fairly destructive sparks. There are not many but so far they have tripped off all optics in ITW, the water flow interlocks (later forced on to prevent this), the EL (generally comes back but killed the canbus card once), destroyed two vacuum gauge controllers, and may have contributed to the cathode short on ITE-TM3-UC44-LP-FEBIAD.

The conditioning process seems to go as planned, but the occassional inevitable spark still trips things, which is making it difficult to run stable.

Will continue with regular HV conditioning to see if this helps. Control over the current limit on the ITE:BIAS supply would also help, but seems a complicated thing to add according to Roberto.

TM3 sat fine at 53kV overnight and at 58kV for several hours. The EZL trips off maybe once every 2-3hrs or so, which is not ideal but liveable. This was with the TGHT and CHT power supplies forced, so not sure if we would know if they would have tripped...

CONCLUSIONS:

1. TM3 has been commissioned for beam delivery after one week of delivery of UF to TITAN using a FEBIAD ion source in anode bias mode. Cathode mode was also used, so all functionality of the target except IGLIS current/voltage delivery through the mutlipin has been tested successfully.
2. TM3 still requires commissioning at high voltage with protons to be considered fully commissioned, this will be done July 2 2024 due to problems with the cyclotron.

 the five shielding blocks have been moved from storage area to ITE and is ready for beam delivery.

submitted WP# I2024-05-22--1

actived  WP # I2024-05-23-4

 TM3 with target UCx#44 was disconnected in ITE. 

Tm3 has been moved form ITE to SHC. the move is smooth.

IG1S elbow o ring seal and a blank off cap o-ring seal beside the elbow were replaced on ITE entrance module top last Friday. IG1s has a higher than normal reading so i leak checked entrance module top last Thursday and found leaks to 5x E-8 atm.cc/sec leak on one of (or both of )these two o-rings seals. they are too close to identify exactly on which one was leaking so both o rings were replaced.  other air to vacuum seal on entrance module are leak free. 

A new gas system for FEBIAD operation was installed in ITE to enable us to use a calibrated leak which is attached to the target heat shield. This will allow us to control the quantity and purity of the gas going into the system, and to switch quickly between gasese and pressures.

The schematic of the system is attached. The use of the ballast container is likely not needed - it was originally included in case we want to run the system at pressures greater than 1 ATM. The roughing pump inlet limit is 1ATM, so it was envisaged to used the ballast container to reduce the pressure to that level or below before opening the system to the pump. But since it has been difficult to commission the system over 1ATM and this opens the door to several concerns about gas leakage, it was deemed better to keep operation below 1 ATM. This tank will likely be removed.

The storage tank was implemented because we are not sure if irradiated gases will be pulled out of the target lines when we pump on them, and so this was devised as a way to test that. The alternative is to have the exhaust of this pump go directly into nuclear ventilation by way of a plastic tube leaving the faraday cage, however this solution is also quite some work to implement. So the consensus was to start with the storage tank and sample it several times to understand what sorts of radioactivity we are dealing with, then make a decision about future operations.

This system has been tested offline using the development target UCx#44 and appears to work well. Unfortunately there were no protons available for this beam time so the gas sampling part of the system was not tested.

The operation of the rest of the system worked reasonably well. Beam disappeared when pumping out the lines and reappeared when re-opening the gas bottle. It seems that the storage container can be used to pump out the lines about 7 times before it gets up to around atmosphere, as was estimated based on volume. The pump can get the lines down to about 250mTorr as read by the pressure gauge DG2 on the module top, given about 30mins of pumping.

We did have some issue with the gauges, HV sparks destroyed both controllers and so we were unable to monitor pressures as we would have liked. It is possible one gauge itself is broken as well, to be investigated.

It may be that the gas leak we used was too small, but the idea of changing the pressure on the module by changing the needle valve didn't seem to work. This is not sure since we didn't have a reading on the pressure, but watching the ion beam as you turn the gas back on seemed quite binary. You would see no change until you had the needle valve fully open. This may be different with a larger leak, since this leak required about 1ATM in the lines to see anything. It is still not clear why this behaviour was different than the tests at the test stand, in which 0.5ATM of upstream pressure produced about 2nA of beam.

This system is considered a prototype and is not turned over to ops for operation. Further commissioning is required.

Tm3 has been moved from TCS to ITE. the move is smooth.

Tm3 has been moved from TCS to ITE. the move is smooth.

 TM3 with new target SiC#46 was connected in ITE.

Initial Summary:

TM3 sparking behaviour in ITE has been very disruptive both with and without protons. Protons are at 60uA, target heater at 540A. Max voltage achieved so far is 51kV, limited by time and a current limit on the BIAS supply.

The ITE:EL trips off every couple of minutes. Just needs to be restarted. Mike investigated and found that the trip is happening inside the power supply, then a moment later the canbus is defining the supply off when it cant get it to respond. Trip frequency decreased over time but still disruptive.

All the waterflow signals, including ones controlled in the cyclotron system, were tripping off regularly. These have all been forced, then we implemented a 5s delay in the ISAC waterflow interlocks. This seems to have solved the issue.

The ITE:BIAS supply has also been limited at 500uA instead of 5mA. Probably something hardcoded into the canbus - this was fixed by Keiko and the hardcode limit was set to 20%(?) of 5mA.

TM3 has been very sparky, even at low voltages. I can't understand why. A new ground path would be explicable but the excessive sparking is difficult to understand.

Possible avenues:

1. make sure the module outer surface is connected through a copper sheet to the sheet on the wall.
2. check that special ground still grounds all the gnd power supplies in the electrical room - this I checked but some of the grounds are connected through wires instead of sheets, this can be improved.
3. try running the module with only HV hooked up and nothing else.
4. try running TM3 in ITW.
5. Search work permits and shutdown jobs for any changes made to the system that could affect this.
6. hope there is a software solution for the EL problem, or start moving towards PLC control - no software solution because the canbus card is only reacting to the power supply refusing to turn on, so it seems the issue originates in the PSU. Moving to PLC is possible but may not help and increases the risk of frying the PLC module.
7. Add a spark gap or limiting resistor to the EL somewhere. Resistor only helpful if the EL itself is sparking. Difficult to find a spot for the spark gap.
8. Get a bipolar supply that can sink the current of a spark and replace the EL supply. - Tomislav doesn't seem to think this is viable.
9. Add protection on the controls cables.

Spark recording:

60uA/50kV/540A July 3 02:30 - 05:30 : average of 50 sparks per hour. data file https://isacwserv.triumf.ca/onlylocal/isacdata/ITE_TM3_SiC46_HVConditioning_2_20240704

60uA/32kV/540A July 3 19:15 - July 3 21:45 : average of 8.4 sparks per hour. data file https://isacwserv.triumf.ca/onlylocal/isacdata/ITE_TM3_SiC46_HPSIS_HVConditioning_20240704

50uA/42.8kV/540A July 4 8pm - July 5 2pm : average of 20 sparks per hour, or if you cut out the last two hour that were very sparky, 14 sparks per hour. data file http://isacwserv.triumf.ca/onlylocal/isacdata/TM3commissioning_July4_52024.dat

50uA/42.8kV/540A July 8 00:00 - July 8 18:00 : average of 3.4 sparks per hour. data file http://isacwserv.triumf.ca/onlylocal/isacdata/TM3commissioning_July82024.dat

50uA/42.8kV/540A July 10 10am - 3pm : 3.6 sparks per hour

50uA/53kV/540A July 17 5am - 10am : 4 sparks per hour

45uA/53kV/540A July 20 8pm - July 21 6:30am : 24 sparks in 11 hrs, avg 2.2 sparks per hour.

EL trip recording (first 2 entries from the archiver, verify with ops elog):

50uA.42.8kV/540A July 4 23:00 - July 5 16:00 : 9 trips, average 0.5 trips per hour

50uA/42.8kV/540A July 6 12:00 - July 7 03:00 : 6 trips with 2.5hrs off, average 0.5 trips per hour

50uA/42.8kV/540A July 7 05:00 - July 8 18:00 : no trips in past 30hrs. Last trip July 7 at 3am.

50uA/42.8kV/540A July 10 3pm - still no EL trips

50uA/53kV/540A July 16 8pm - July 17 10am : 3 trips in 6 hrs. Note cable in EL circuit got burned so EL is running at only ~4kV

45uA/53kV/540A July 17 3pm - July 18 9:30am : 7 trips in 18hrs

45uA/53kV/540A July20 14:00 - July 21 6:30 : 5 trips in ~17hrs

2024-11-15 EDIT, Alexander Shkuratoff:

I was not involved in this run, but I have gone through all the Elogs for TM3 up to this point and summarized all the data dumps at the bottom, so I may as well do it here.

	ITE_TM3_SiC46_HPSIS_HVConditioning_20240704	2024-07-04 01:42	4.8M
	ITE_TM3_SiC46_HVConditioning_Vac_20240704	2024-07-04 01:59	5.3M
	ITE_TM3_SiC46_HVConditioning_2_20240704	2024-07-04 05:31	1.1M
	TM3commissioning_July32024.dat	2024-07-04 17:04	11M
	TM3commissioning_July4_52024.dat	2024-07-05 14:18	12M
	TM3commissioning_July82024.dat	2024-07-08 18:03	13M

 Additionally, there is no Elog entry for August 18, 2024 for which TM3 commissioning continued in ITW, but I have found the following StripTool data:

	TM3commissioning_Aug182024.dat	2024-08-18 05:37	14M
	TM3commissioning_Aug182024_2.dat	2024-08-19 06:16	14M

ITE:EL has been tripping off due to HV sparks. Usually this also trips ITE:Q1/2. These are sitting in side-by-side racks.

The working theory is that HV sparks are returning to ground through the EL and quads, tripping them off. Possibly this is the result of a changed low impedance ground path. How this change happened could be either something we did during shutdown or the changes we made in TM3, no clear evidence yet.

The goal of these changes is to stop the EL and quads from tripping, assuming the sparks are travelling through the ground.

Possible routes through ground to the EL power supply:

1. Through controls. EL is controlled by a canbus card. Preliminary investigations by Mike Rowe make it look like the canbus reads the PSU max voltage (60kV), then the staton bit trips, then after a while of trying to talk to the power supply, the statdrv is turned off. It is not clear why the power supply is turning itself off, since it doesn't seem to have a trip or protection circuit inside. To be further investigated.
2. Through the AC ground.
3. Through the shield on the output voltage conductor that runs to the pit.
4. Through the EL wire and conductor itself.

Current attempts to mitigate the tripping:

• Add an isolation step to the AC power, in case it is coming through AC ground (option #2).
• Add ferrites on the controls lines (option #1, this was done earlier and no effect has been seen).
• Add transient voltage suppresors on the Vmon, Imon, Vprog and Iprog control cables (option #1).
• Replace grounding cable with a grounding strip where the grounding wire used to be on ITE:EL. The ground wire was a braid, so probably not too bad, so not sure if this will help.
• Improve the connection between the two grounding bars in the racks. The grounding bar for th EL supply is well connected to the copper ground strip from ITE, but only connected through a small cable to the grounding bar for the quads. A copper foil sheet was added between them and the previously used wire was disconnected on the quad side bar..

 TM3 with spent target SiC#46 was disconnected in ITE and ready to move to TCS soon.

 TM2 with new target Ta#67 was connected in ITE, all TP are running at their full speed now.

TM2 with Ta#target has been moved from SHC to ITE. the move is smooth.

 TM2 with spent target Ta#67 was disconnected in ITE. see attachment pls.

Tm2 has been moved form SHC to ITE. The move is smooth.