The TiC#5 post irradiation inspection was done on Wed. June 14 and Mon. June 19. This target didn't have any obvious issues but it didn't produce much beam. The inspection revealed that the container was quite embrittled and showed significant damage, though it was not broken. The FEBIAD grid looked fine, despite the large anode current draw (indicating that current was flowing elsewhere). The gas line looked fine though the gas injection to this target looked as if it wasn't connected. But there was a plug on the unused line, so it must have been correct. No other obvious signs that would explain the poor performance. Pictures are can be found here https://documents.triumf.ca/docushare/dsweb/View/Collection-37913

Looked at SiC#45 in the hot cell. Looked very clean, no deposits on the heat shield or other components. Target also looked very straight. Only issue we saw was sparking marks on the EE side that faces the ground electrode, in several spots. This target had a bout of sparking when ramping up the protons, so I guess that was the location. Will look for matching spark marks on the ground electrode next time TM4 is in the hot cell. Forgot to take picutres...

PIE on TiC#6 was done in the NHC. The target assembly looked brand new when we took it out, no heating spots or deposits (pictures on docushare). Online we measured a 400kOhm connection between TGHT-TBHT-COIL to HV common (TGHT and COIL shorted in TM2 service tray) and anode was unstable above ~300V. After the module move and before the target removal David found TGHT-TBHT-COIL was a dead short to HV common and the anode was at 39kOhms to HV common.

1. Wanted to see if the TGHT-TBHT-COIL short was on the COIL. Measured with ohmmeter the COIL lines to HS and got about 700Ohms. But with the target lying down the COIL lines drop and touch the heat shield so to do this measurement Chad had to lift the lines, which could cause them to touch something else. So not sure how reliable this measurement was. It jumped around a lot, saw 0L occasionally.

2. Measured both target legs and tantalum gas line to HS, got about 5kOhms repeatedly using the ohmmeter.

3. Meggered across the gas line insulator to the gas line going into the target and got open circuit at 250V, so the gas line wasn't coated.

4. Target container looked vaguely red in the areas where the target material was, otherwise still no deposits on the inside. The container was not overly crystalized as seen on the legs and end caps.

5. Some sparking marks on the EE and some slightly asymmetric deposits

Continuing Aug. 22 2023:

6. Found what looks like a hole in the insulation on the COIL, maybe the cause of our short? Found a corresponding line mark on the copper plates that hold the coil down but not sure if it is related. Pics on docushare.

7. Grid looks perfectly new. Where was that current going?

8. Copper plates that hold coil were loose and bolts that hold coil in just fell out as well.

9. What can be seen of the anode wire looks good.

Pumped down the gas lines in ITE on Tuesday Nov. 28 2023 and opened the CF4 bottle valves. Missed the small section between the regulator and SV2 though, so will probably redo it.

Pumped down again on Dec. 12 2023. Was taking a long time so might not be as good as earlier. Maybe there is a leak in there...

Tuesday, January 02, 2024, 15:58 :

Started HV ramp up on TM3 service tray about 3pm today. Had to bypass TCS:BIAS interlocks to make it work - there are some flow restrictions on the water lines that are causing the flow interlock to periodically trigger. The water flow it not very important for this test as long as the water is moving. Turned out that it was also conveninent to have all the interlocks bypassed because sparks tripped off the IG1 gauge and this did not cause the HV bias supply to turn off. Should use same setup next time.

Some sparking around 35-44kV, tripped off turbos/IG1/IG1S 3 times and so decided to ramp down.

Tried again while standing outside TH door to hear if sparking is outside the vacuum. Went to 48kV very slowy with multiple trips of the turbos, mostly TP2. Lots of sparks but easy to ramp back up to previous high point, so likely just a painful conditioning process. Current draw 167uA at 48kV. No sparks audible outside door, and vacuum response looks too large to be electrical noise.

Continued on Wednesday January 03 - went up to 53kV in the same way as before, lots of TP2 trips making its current a bit unstable... data was dumped as TM3_servicetray_conditioning_jan32023. Tried limiting the current on the power supply to very near the drain current to see if that reduces the number of trips. Didn't notice a significant effect.

Thursday, January 04, 2024, 09:57 :

Restarting conditioning, starting at 180uA at 53kV. Got up to around 60kV with many trips of the turbos, and one restart of the IG1 controller needed. Added a grounding hook to the vacuum controller rack to see if that helps reduce the trips. (doesnt appear to have).

Friday, January 05, 2024, 09:51 :

Continuing conditioning. Starting by sitting at 50kV for 10mins. No sparks observed during this time.

Even though we got to 60kV last time, onset of conditioning started at 58.6kV with a trip at 58.9kV... got up to 70kV.

Got to 72kV on Monday January 08 2024.

On Monday Jan. 15, tried to improve the grounding on the vacuum services rack to reduce sparking through the turbo controllers. Added two welding cables from the vacuum rack ground busbar to the copper grounding strip on the ground. If the spark is travelling from the faraday cage to the rack through conduit or grounding cables, maybe this will help. If it is coming through the turbo cable then it likely won't. Tried to add grounding straps on the turbo controller chassis (there is a brass screw which I think is for this purpose) but couldn't get either of the screws out with any resonable force. They undo a bit then get stuck...

 Wednesday, January 17, 2024, 20:39:

David vented TM3 with N, then opened the service cap to install new insulators. We pumped back down and Alexander and I ramped back up today - instabilities started around 55kV and we were back to slow conditioning in the high 50s. Got up to a set point of 65kV. Ray did some work on the TCS bias supply and now neither the set point or readback is correct. The PSU was reading 51.5kV at set point 50kV and 60kV at set point 58kV. Will continue later...

Wednesday, January 31, 2024, 16:23

Ramped up again, some slight instabiities around 56kV but could still go up pretty quick until about 65kV. Since Ray's testing of the TCS:BIAS supply both the readback and the set point are off from the powersupply display. Ran out of time and ramped down at set point 70.5.

Tuesday, February 06, 2024, 14:47

Testing again after venting the module with air. First tiny spark at 48, then some vacuum reactions starting at 49 but still no issues. Bit more reactive around 53.

 stopped at 69.4

Wednesday, February 07, 2024, 15:28 :

Ramped easily to 60kV set point and let it sit there. Didn't spark at all then after 100mins, sparked TP2 off. Otherwise first sparking happened at set point 67.

Found at limit at around 72.5 on the PSU front output, set point 70.6. think it comes from EPICS. Will have to test the power supply.... stopped there.

Thursday, February 08, 2024, 15:03 :

Long term testing. ramped back up, first spark at set point 65.

Let it sit for 2hrs 20mins. Something like 6 sparks during this time. Current also dropped significantly from 0.32A to 0.27A.

EDIT 2024-11-25, Alexander Shkuratoff:

All the StripTool data related to the above tests are on the server as the following below. Those without "Alexander" in the name were dumped by Carla.

	TM3_servicetray_conditioning_Jan32023	2024-01-03 17:29	807K
	20240103_TM3_Service_Tray_TCS_Alexander	2024-01-03 17:31	1.2M
	20240104_TM3_Service_Tray_TCS_Alexander	2024-01-04 16:58	238K
	20240104_TM3_Service_Tray_TCS_Alexander_2.csv	2024-01-05 07:00	13M
	20240104_TM3_Service_Tray_TCS_Alexander_1sec.csv	2024-01-05 12:21	81
	20240105_TM3_Service_Tray_TCS_Alexander.csv	2024-01-05 16:51	13M
	20240105_TM3_Service_Tray_TCS_Alexander_1sec.csv	2024-01-05 16:52	2.7M
	TM3_servicetray_conditioning_Jan52023	2024-01-05 21:24	8.8M
	20240108_TM3_Service_Tray_TCS_Alexander_1sec.csv	2024-01-08 16:56	4.7M
	20240108_TM3_Service_Tray_TCS_Alexander.csv	2024-01-08 16:56	7.1M

	TM3_servicetray_conditioning_Jan172024	2024-01-17 13:29	257K
	20240117_TM3_Service_Tray_TCS_Alexander.csv	2024-01-17 20:11	4.0M

	TM3_servicetray_conditioning_Jan312024	2024-01-31 17:24	2.2M
	20240131TM3_Service_Tray_TCS_Alexander.csv	2024-01-31 17:55	4.5M
	20240206_TM3_Service_Tray_TCS_Alexander.csv	2024-02-06 17:58	2.6M
	TM3_servicetray_conditioning_Feb62024	2024-02-06 19:29	4.6M
	TM3_servicetray_conditioning_Feb82024	2024-02-08 17:55	3.1M

 And the StripTool settings:

Alexander_TCS_HV_Conditioning.stp

2024-01-17 19:57

3.7K

 To deliver stable beam to RadMol, started heating UCx#43 up again. Target was overheated during initial conditioning in Nov but then ran well for 4 weeks. During the run time, the TGHT voltage jumped around and this was reflected in the ion current, but yields seemed unaffected.

When re-heating it, the TGHT voltage was very unstable and once it got to 440A, it went into voltage limiting mode. We ramped down and it was still voltage limiting, but turning things off solved it. We ramped back up and it did about the same thing. TBHT voltage and current remained steady. Looks like a heat generated issue.

Measured the resistance in the lines from the electrical room. Resistance of the lines to ITE should be 1.19mOhms for TGHT and 1.95mOhms for TBHT. TBHT positive to negative had a resistance of 7.9mOhms from the electrical room today, and 5.7mOhms on module top when measured Nov. 8. This seems to track.

TGHT positive to negative was measured at 27.6mOhms today (and was jumping around), whereas it was 7.14m Ohms on Nov. 8.

10V @ 440A is 22.7mOhms, so it makes sense that it limited. Tried with increased voltage (up to limit of 16V) but this didn't help. Declared the target dead on Jan. 11.

David measured resistances on the module top and found (in mOhms):

A-B = 27 (TGHT)

A-C = 7.473

A-D = 6.380

B-C = 26.053

B-D = 25.569

C-D = 5.212 (TBHT)

So looks like the tube is broken on the B side.

On Jan. 9 we did the PIE on Ta#66. Opened the heat shield and removed the EE. The inside of the heat shield was very black with lots of flakes falling off. The target itself looked overheated, and the container looked pretty bad for a Ta target. The outside of the EE looked perfect but after removing it there was a lot of black stuff on the side and we saw that the Re foil was curled up. Jens said during the runtime that he saw something that looked like a wire in the beam path, so maybe this was it.

UCx#43 PIE was done in the SHC. As expected based on electrical measurements, the target container was broken on the right side and came off easily. Despite all the black stuff seen on the containment box and source tray, the inside of the heat shield was very clean. Checked if the Re foil was curled in and didn't see anything. Looked fine other than the broken container.

ITW was set up for HV testing - water is running and at a high resistivity due to injected N. Einzel lens is attached to ground. Everything else is as normal. The module plug is in so no vacuum bypasses were needed. Very easy to ramp to 60kV. Sat at 60 for about 1.5hrs with maybe 1 spark (hard to say from the strip tool data). Current draw was about 176uA. Data dumped as isac/data/ITW_HVtesting_shutdown2024

ITE was set up for HV tests - this time the module plug was not in so the BIAS and EE had to be bypassed. NOTE: ITE:BIASPLC must also be bypassed for this to work. Currently it is located on the vacuum bypasses page, but may be changed to the optics page soon.

ITE was ramped to 60kV in steps of 1kV easily. It sat at 60 for about 1.5 hours with a current draw around 173uA. It was not entirely stable though, and appears to have sparked many times. None of them seemed very large, but hard to tell.

When ramping down, current dropped quickly with the first couple steps. Data saved as isac/data/ITE_HVtesting_shutdown2024

Wires were pulled in TM3 and service cap was closed up again:

Einzel lens wire (30kV kapton), routed in the pumping duct and attached with #10 ring tongue to EZL feedthrough in service cap

4 steerer wires, routed in th pumping duct and attached with push on connectors to the 4 SHV feedthroughs on the module side panel. Coax grounding sheath is cut back by about 3cm.

Multipin cable bundle #1, routed through the left cable tube in the service tray then along the heat shield water lines (with PEEK zip ties) to feedthrough A.

Multipin cable bundle #2, routed through the right cable tube in the service tray then along the heat shield water lines (with PEEK zip ties) to feedthrough B. This bundle contains the EE wire as well, which was branched out after exiting the service tray and routed along the gas lines to the EE feedthough. It was attached with a smaller (size?) ring tongue to the EE feedthrough. Do not use this wire again, very fragile. Use the Accuglass 30kV wire instead.

The multipin wires touch the back of the water blocks in the source tray (same as in MT2/4, so far no issues with that) and they also touch the COIL lines in the service cap.

The EZL and steerer wires are routed along the floor of the service cap.

Due to the difficulty of the installation, there is no metal braid over the multipin bundles in the service cap.

Photos here : https://triumfoffice365-my.sharepoint.com/:f:/g/personal/cbabcock_triumf_ca/EslWNUST-UJKh1s8jjxWG6EBpLQEnWxgV4v-NvR4ZJjlOQ?e=Mw5vvI

ITW gas lines pumped down to 2e-3Torr on leak detector with leak rate in the low e-7 atm-cc/s. Gas bottles opened at 3psi and labels in epics changed to reflect correct setup. switch box set to cathode mode, checked that anode wire was at HV common.

note ballast is removed from ITW now to be used in ITE so that line is blanked off.

Tuesday, May 14, 2024, 10:26 :

Ramping up bias/EE/EL on ITE - will see if the HV cage in the pit can hold 60kV or not.

At around 50kV the bias read current starts to get jumpy. Doesn't look like its an issue for the voltage, but definitely instability increases.

Occasional sparks seem to increase the EL current significantly...

David heard some sparks in the target hall. Looks like from heat shield feed through to the top of the cage. Several sparks heard in the TH also changed the einzel lens draw current significantly. These feethroughs are not far from each other so could be related.

Apparently the sparking also tripped off all the optics and heaters in ITW, by changing the state of the water flow interlock. Interesting and not seen ever before. Will test again with the cage off, rather than risk tripping ITW again.

Tried ramping up the voltage on just the EE. Seems the limit is 7kV - at that level it gets sparky and unstable. Should be investigated (should hold 10kV) but not critical for module function.

Wednesday, May 15, 2024, 10:05 :

HV cover was removed this morning and the MAA was locked out.

Started ramping things up, was very easy until around 58kV. At 59.5kV a spark tripped off the EL. Canbus controller had to be power cycled to fix it. Bias current is much more stable in this configuration.

Ramped up again while listening from the TH to see if it was sparking outside. Went slowly to 60kV (left EL at 29kV) and didn't see any outside sparks in the first 10 mins, couple sparks in vacuum. But after 15 mins a spark seems to have fried the EL canbus card. We were not in the TH so not sure if it was on the outside or not.

After 2 hours a spark tripped off both ITE and ITW. Looks like it makes a water interlock go bad momentarily, but after the trip the interlocks seem fine.

Tuesday, May 28, 2024, 16:41

Running TM3 at any voltage in ITE produces fairly destructive sparks. There are not many but so far they have tripped off all optics in ITW, the water flow interlocks (later forced on to prevent this), the EL (generally comes back but killed the canbus card once), destroyed two vacuum gauge controllers, and may have contributed to the cathode short on ITE-TM3-UC44-LP-FEBIAD.

The conditioning process seems to go as planned, but the occassional inevitable spark still trips things, which is making it difficult to run stable.

Will continue with regular HV conditioning to see if this helps. Control over the current limit on the ITE:BIAS supply would also help, but seems a complicated thing to add according to Roberto.

TM3 sat fine at 53kV overnight and at 58kV for several hours. The EZL trips off maybe once every 2-3hrs or so, which is not ideal but liveable. This was with the TGHT and CHT power supplies forced, so not sure if we would know if they would have tripped...

CONCLUSIONS:

1. TM3 has been commissioned for beam delivery after one week of delivery of UF to TITAN using a FEBIAD ion source in anode bias mode. Cathode mode was also used, so all functionality of the target except IGLIS current/voltage delivery through the mutlipin has been tested successfully.
2. TM3 still requires commissioning at high voltage with protons to be considered fully commissioned, this will be done July 2 2024 due to problems with the cyclotron.

On May 30 Aaron, Chad and I removed the empty LP FEBIAD target from the SHC into the anteroom to attempt to find out the source of the cathode short observed online. The behaviour online was that a connection of several hundred ohms was preventing the cathode from being biased. The resistance was still enough to allow us to run in anode mode, which is what we did. I am not sure where the cathode short came from but there was significant sparking on this module when ops conditioned it (destroyed the gmov of the COIL power supply and the electronics for anode/cathode switching) and we have seen this cause cathode shorts before. The shorted behaviour did not go away when the target was cooled down, so I assume it was permanent damage to one of the insulators. This would be either the support insulator, the trident insulator, the gas line insulator or one of the tube retainer insulators.

Since the target didn't see protons, we decided to take it out of the hot cell and disassemble it to find which insulator was broken. Unfortunately, it appeared totally fine outside of the hot cell, no shorts could be found. We disassembled it anyway to look for issues but all we saw was a heat mark on the copper plates covering the coil, underneat the target (see attachment). We even shook the target quite a bit, and didn't produce a short.

Maybe there was a trace created on one insulator by the sparking, which oxidized when exposed to air and became insulating again? We didn't see anything obvious on the insulators, but I'm not sure if we would expect to.

No conclusions from this work.

ITW has been behaving badly at higher voltages recently, even without protons on. I conditioned it up to 50kV with heaters off in the TCS and it was fine. But when I tried to do the same with heaters on in the station, it was quite sparky and it kept tripping the TGHT/TBHT. I went to 49kV where it was unstable, then lowered to 48kV where it looked fine for 30mins. But even operating at 40kV, it tripped the heaters several times. The TBHT voltage came back lower than before on the last trip, so I am getting worried about it.

I decided to make sure all the electronic spark protection was ok in the faraday cage. There is a resistor/gmov combination across the terminals of each TGHT/TBHT in which Tomislav replaced the gmovs for me because his tester did not work at those low voltages, so we couldnt see if they were broken. The electronics diagram called for 18V gmovs, the replacements are 22V because that's all we have.

We also tested and re-installed the capacitor-gdt-resistor combinations that John installed last year from TGHT/TBHT bubars to HV common. I originally removed them because we couldn't bias a FEBIAD with them on - this still has to be investigated. The measured values are 0.148uF, 90kOhms and a breakdown voltage on the gdt of 410-450V.

Hopefully this prevents sparking the power supplies off.

Sunday, June 16, 2024, 07:03 : power supplies have not tripped off since electronics repair. See data/TM4sparkingbehaviour_June152024.dat for sparking frequency over ~10hrs of protons (5-14uA). >14 sparks during this time (14 captured by PLC when voltage dropped, many more current peaks).

A new gas system for FEBIAD operation was installed in ITE to enable us to use a calibrated leak which is attached to the target heat shield. This will allow us to control the quantity and purity of the gas going into the system, and to switch quickly between gasese and pressures.

The schematic of the system is attached. The use of the ballast container is likely not needed - it was originally included in case we want to run the system at pressures greater than 1 ATM. The roughing pump inlet limit is 1ATM, so it was envisaged to used the ballast container to reduce the pressure to that level or below before opening the system to the pump. But since it has been difficult to commission the system over 1ATM and this opens the door to several concerns about gas leakage, it was deemed better to keep operation below 1 ATM. This tank will likely be removed.

The storage tank was implemented because we are not sure if irradiated gases will be pulled out of the target lines when we pump on them, and so this was devised as a way to test that. The alternative is to have the exhaust of this pump go directly into nuclear ventilation by way of a plastic tube leaving the faraday cage, however this solution is also quite some work to implement. So the consensus was to start with the storage tank and sample it several times to understand what sorts of radioactivity we are dealing with, then make a decision about future operations.

This system has been tested offline using the development target UCx#44 and appears to work well. Unfortunately there were no protons available for this beam time so the gas sampling part of the system was not tested.

The operation of the rest of the system worked reasonably well. Beam disappeared when pumping out the lines and reappeared when re-opening the gas bottle. It seems that the storage container can be used to pump out the lines about 7 times before it gets up to around atmosphere, as was estimated based on volume. The pump can get the lines down to about 250mTorr as read by the pressure gauge DG2 on the module top, given about 30mins of pumping.

We did have some issue with the gauges, HV sparks destroyed both controllers and so we were unable to monitor pressures as we would have liked. It is possible one gauge itself is broken as well, to be investigated.

It may be that the gas leak we used was too small, but the idea of changing the pressure on the module by changing the needle valve didn't seem to work. This is not sure since we didn't have a reading on the pressure, but watching the ion beam as you turn the gas back on seemed quite binary. You would see no change until you had the needle valve fully open. This may be different with a larger leak, since this leak required about 1ATM in the lines to see anything. It is still not clear why this behaviour was different than the tests at the test stand, in which 0.5ATM of upstream pressure produced about 2nA of beam.

This system is considered a prototype and is not turned over to ops for operation. Further commissioning is required.

TM3 and UCx#44 ran in ITE but issues with the cyclotron prevented proton beam delivery. Nonetheless, the target was heated and provided stable beam to TITAN as well as for tuning and some tests.

Results of inspection of TM3 module after run:

The module looked pretty good. There were no marks on the polished surface of the containment box, next to the water blocks, that were visible. But we didn't remove the box, only looked with the camera and it was visible by turning the module. So some things may have been hidden. There were some black marks on the door, near the bolts that hold in the handle (see attachment). This seems a very unlikely spot for sparking as the fields should be considerably lower than at the target handles, lower down. Maybe the spots are from oil left on the door that had a chemical reaction? There is one very faint matching spot on the top of the copper trident connector, but it doesn't look much like a spark mark. The other two marks have no corresponding mark on the source tray. There were no marks on the ground electrode, or anywhere else on the source tray. There was a scuff on one of the trapezoidal insulators, but it could have happened during installation.

Results of inspection of the target after run:

The FEBIAD target developed the same cathode short we have seen on almost every FEBIAD we run online - a high resistance short from cathode to HVC that prevents voltage on the cathode. This time the short remained after cooling the target, and David measured 208kOhms with the ohmmeter on the module top (11kOhms with the megger). We used a multimeter in the hot cell to test after the target had been dismounted and we found no short. So it must be related to either the position of the target (upright vs face down) or time (a conductive trace on an insulator oxidizes and becomes non-conductive again) or some pressure/shaking during the de-installation.

There weren't many marks on the target - a faint mark near where the faint mark on the copper trident was observed, some marks on the EE which had no counterparts on the ground electrode. See pictures.

Initial Summary:

TM3 sparking behaviour in ITE has been very disruptive both with and without protons. Protons are at 60uA, target heater at 540A. Max voltage achieved so far is 51kV, limited by time and a current limit on the BIAS supply.

The ITE:EL trips off every couple of minutes. Just needs to be restarted. Mike investigated and found that the trip is happening inside the power supply, then a moment later the canbus is defining the supply off when it cant get it to respond. Trip frequency decreased over time but still disruptive.

All the waterflow signals, including ones controlled in the cyclotron system, were tripping off regularly. These have all been forced, then we implemented a 5s delay in the ISAC waterflow interlocks. This seems to have solved the issue.

The ITE:BIAS supply has also been limited at 500uA instead of 5mA. Probably something hardcoded into the canbus - this was fixed by Keiko and the hardcode limit was set to 20%(?) of 5mA.

TM3 has been very sparky, even at low voltages. I can't understand why. A new ground path would be explicable but the excessive sparking is difficult to understand.

Possible avenues:

1. make sure the module outer surface is connected through a copper sheet to the sheet on the wall.
2. check that special ground still grounds all the gnd power supplies in the electrical room - this I checked but some of the grounds are connected through wires instead of sheets, this can be improved.
3. try running the module with only HV hooked up and nothing else.
4. try running TM3 in ITW.
5. Search work permits and shutdown jobs for any changes made to the system that could affect this.
6. hope there is a software solution for the EL problem, or start moving towards PLC control - no software solution because the canbus card is only reacting to the power supply refusing to turn on, so it seems the issue originates in the PSU. Moving to PLC is possible but may not help and increases the risk of frying the PLC module.
7. Add a spark gap or limiting resistor to the EL somewhere. Resistor only helpful if the EL itself is sparking. Difficult to find a spot for the spark gap.
8. Get a bipolar supply that can sink the current of a spark and replace the EL supply. - Tomislav doesn't seem to think this is viable.
9. Add protection on the controls cables.

Spark recording:

60uA/50kV/540A July 3 02:30 - 05:30 : average of 50 sparks per hour. data file https://isacwserv.triumf.ca/onlylocal/isacdata/ITE_TM3_SiC46_HVConditioning_2_20240704

60uA/32kV/540A July 3 19:15 - July 3 21:45 : average of 8.4 sparks per hour. data file https://isacwserv.triumf.ca/onlylocal/isacdata/ITE_TM3_SiC46_HPSIS_HVConditioning_20240704

50uA/42.8kV/540A July 4 8pm - July 5 2pm : average of 20 sparks per hour, or if you cut out the last two hour that were very sparky, 14 sparks per hour. data file http://isacwserv.triumf.ca/onlylocal/isacdata/TM3commissioning_July4_52024.dat

50uA/42.8kV/540A July 8 00:00 - July 8 18:00 : average of 3.4 sparks per hour. data file http://isacwserv.triumf.ca/onlylocal/isacdata/TM3commissioning_July82024.dat

50uA/42.8kV/540A July 10 10am - 3pm : 3.6 sparks per hour

50uA/53kV/540A July 17 5am - 10am : 4 sparks per hour

45uA/53kV/540A July 20 8pm - July 21 6:30am : 24 sparks in 11 hrs, avg 2.2 sparks per hour.

EL trip recording (first 2 entries from the archiver, verify with ops elog):

50uA.42.8kV/540A July 4 23:00 - July 5 16:00 : 9 trips, average 0.5 trips per hour

50uA/42.8kV/540A July 6 12:00 - July 7 03:00 : 6 trips with 2.5hrs off, average 0.5 trips per hour

50uA/42.8kV/540A July 7 05:00 - July 8 18:00 : no trips in past 30hrs. Last trip July 7 at 3am.

50uA/42.8kV/540A July 10 3pm - still no EL trips

50uA/53kV/540A July 16 8pm - July 17 10am : 3 trips in 6 hrs. Note cable in EL circuit got burned so EL is running at only ~4kV

45uA/53kV/540A July 17 3pm - July 18 9:30am : 7 trips in 18hrs

45uA/53kV/540A July20 14:00 - July 21 6:30 : 5 trips in ~17hrs

2024-11-15 EDIT, Alexander Shkuratoff:

I was not involved in this run, but I have gone through all the Elogs for TM3 up to this point and summarized all the data dumps at the bottom, so I may as well do it here.

	ITE_TM3_SiC46_HPSIS_HVConditioning_20240704	2024-07-04 01:42	4.8M
	ITE_TM3_SiC46_HVConditioning_Vac_20240704	2024-07-04 01:59	5.3M
	ITE_TM3_SiC46_HVConditioning_2_20240704	2024-07-04 05:31	1.1M
	TM3commissioning_July32024.dat	2024-07-04 17:04	11M
	TM3commissioning_July4_52024.dat	2024-07-05 14:18	12M
	TM3commissioning_July82024.dat	2024-07-08 18:03	13M

 Additionally, there is no Elog entry for August 18, 2024 for which TM3 commissioning continued in ITW, but I have found the following StripTool data:

	TM3commissioning_Aug182024.dat	2024-08-18 05:37	14M
	TM3commissioning_Aug182024_2.dat	2024-08-19 06:16	14M

ITE:EL has been tripping off due to HV sparks. Usually this also trips ITE:Q1/2. These are sitting in side-by-side racks.

The working theory is that HV sparks are returning to ground through the EL and quads, tripping them off. Possibly this is the result of a changed low impedance ground path. How this change happened could be either something we did during shutdown or the changes we made in TM3, no clear evidence yet.

The goal of these changes is to stop the EL and quads from tripping, assuming the sparks are travelling through the ground.

Possible routes through ground to the EL power supply:

1. Through controls. EL is controlled by a canbus card. Preliminary investigations by Mike Rowe make it look like the canbus reads the PSU max voltage (60kV), then the staton bit trips, then after a while of trying to talk to the power supply, the statdrv is turned off. It is not clear why the power supply is turning itself off, since it doesn't seem to have a trip or protection circuit inside. To be further investigated.
2. Through the AC ground.
3. Through the shield on the output voltage conductor that runs to the pit.
4. Through the EL wire and conductor itself.

Current attempts to mitigate the tripping:

• Add an isolation step to the AC power, in case it is coming through AC ground (option #2).
• Add ferrites on the controls lines (option #1, this was done earlier and no effect has been seen).
• Add transient voltage suppresors on the Vmon, Imon, Vprog and Iprog control cables (option #1).
• Replace grounding cable with a grounding strip where the grounding wire used to be on ITE:EL. The ground wire was a braid, so probably not too bad, so not sure if this will help.
• Improve the connection between the two grounding bars in the racks. The grounding bar for th EL supply is well connected to the copper ground strip from ITE, but only connected through a small cable to the grounding bar for the quads. A copper foil sheet was added between them and the previously used wire was disconnected on the quad side bar..

With the new TiC#8 target installed on TM3, there was a ~7kOhm connection between TGHT/TBHT (A/B/C/D) and HV common. The standard electrical check on the module showed standard values, so it was assumed the problem was inside the target.

Started removing bolts/water connections one by one to see if any of them had an effect on the measurement. Found that disconnecting the multipin on the target from the mating connector on the source tray resolved the issue. Determined that the only way this could happen is if a spark cut through the insulation covering the multipin wires. Because of the way they are routed, they can touch the water lines and they are definitely touching waterline C, TBHT-. We do not typically check continuity of multipins to all water lines, just to HVC, so did not see this in the standard electrical check. Measurements verified that pin 2-4 was touching water line C, creating a short from TGHT/TBHT to HV common when the multipin connector is attached, since the pins are shorted to HVC inside the connector when they are not used.

Issue was solved by removing the target into the ante room, removing connector 1 all together since it was not used, and removing the grounding on the other pins, leaving only pin 2-2 connected for the anode.

In future we will change the IGLIS wiring so pin 2-4 is not used, and redesign the cable routing so it cannot touch the blocks. For the immediate term, we will add to the electrical check a check of each pin to TBHT/TGHT/COIL.