Did the PIE on UCx#39-IGLIS on Thursday Nov. 2 2022, couple pictures uploaded to docushare. Did not remove the target, just the heat shield cover.

Final electrical check on the module showed a dead short from TGHT/TBHT to HV common, which we did not see online. Seems like it must have occurred after ramp down? It persisted on the target after we removed it from the module. We couldn't find any obvious contact points, because it seems it did not happen while current was running in the target. Possible locations are the bottom support leg for the TBHT and the top of the right side target leg, though it was difficult to confirm with the camera.

Otherwise the only thing to note was a blacker-than-usual spot on the inside of the Cu heat shield, downstream in the proton direction.

The pin on TM4 has become loose again, making it very difficult to remove the multipin connector with the last target, C#4. For the installation of the next target, UCx#41, we will drill out the hole on the target side bracket so that it doesn't contact the multipin pin. Will try to remove this pin by force in January, or if needed change out the whole multipin bracket.

Post irradiation inspection of C#4 which was running in TM4/ITE was conducted at the NHC on Nov. 22 2022.

The yields from this target had been dropping and its behaviour was erratic, with the voltages on the tube and target jumping all over the place sometimes. The ion current on the FCs followed the voltages. We took this to mean the target container was breaking down.

Upon inspection we found the container looked pretty damaged - highly crystallized and distorted. The container had gone banana-shaped, but not in the usual direction. It had curved  in the direction of the ion source (see photos). The heat shield and other surrounding components were super clean, no deposits or marks at all.

Feb. 16 2023: HV tests done on ITE station. Ramped up to 59kV but was not stable there, current draw was very unstable and was sparking every ~6mins. Ramped down to 58kV and it was stable there for 3 hours drawing 285uA.

Feb. 24 2023: HV tests done on ITW station. Ramped easily to 60kV with no sparking - suspicious. Has been stable at 60kV for 45mins with no sparks and current draw of 200uA. Cannot find any reason that things aren't connected properly...

On Sept. 14 2022, a 10kOhm 30W resistor was installed in each target station faraday cage, in line with the HV bias supply. This was an effort to reduce the current draw from the BIAS supply and protect the power supply controls cards which have been repeatedly breaking due to HV sparks (we think). In the end we still broke cards after the installation, so it doesn't seem to have helped. See below for details on the electrical side:

At 30kV, HV currents are around 100-150uA, meaning the voltage drop over the resistor is around 1.5V. When operating at higher voltages, with higher current draw, the voltage across the resistor could be over 2V and we could get into a situation where it would have an effect outside the normal adjustments needed on the mass separator. We made this decision based on the fact that the power supply's setting resolution is on the order of 1 x 10^-4.

When the resistance of the water changes, this affects the current draw from the power supply and this in turn affects the voltage over this resistor. Based on some sample numbers of running times, there is not much variation in the current except from sparks and if there is some issue, like water leaking between HV and ground. During normal running, the voltage change across the resistor should be less than 0.5V during normal operation. This is on the same order as the 8 hour stability of the power supply.

Yesterday and today module conditioning was done on Tm4 in ITE. This is the first ramp up after the shutdown and ITE is planned to run at high voltages for the first weeks. The module is behaving well - went up to 40kV yesterday, stayed there for an hour, and up to 46kV today. Current draw is 113uA at 46kV. No serious sparking.

Tuesday, May 23, 2023, 09:32 :  Ramping up again, easy to 45kV. Drawing 132uA at 49kV. Still behaving quite well.

When I tried to raise the anode voltage in cathode bias mode on May 23 evening, I found the target or tube shorted to HV common. Investigations in the faraday cage today showed the same. I removed all the new electronics (resistors, gemovs, etc) from the power supply to no effect, so I disconnected the TGHT supply and pushed it away from the busbars. There was still a short. David Frank and I then went down to the pit and disconnected everything to measure the module. The module itself showed no short, and the bus bars in the chase for TGHT showed no short, though the ones for TBHT still showed the short.

David and Frank reconnected the module and I disconnected the TBHT supply in the faraday cage, then I found no shorts to TGHT/TBHT. When I reconnected everything there were still no shorts. There must have been a short in the TBHT line in the faraday cage, and another short in the TGHT line in either the faraday cage or on the module top. Both these shorts got resolved by taking everything apart. Still not sure how it happened.

Next I found I could not raise the anode voltage above 230V, it looked like it was current limiting but since the current limit was high, I figured it had something to do with the new electronics installed. I disconnected them all the in cage and that solved the problem, and then reconnecting the gemovs that go from the PSU terminals to HV common caused the problem again, so I left those disconnected.

Conditioned TM2 up to 48kV in ITW. Some sparks and current is not very stable. Drawing about 178uA at 48kV.

The TiC#5 post irradiation inspection was done on Wed. June 14 and Mon. June 19. This target didn't have any obvious issues but it didn't produce much beam. The inspection revealed that the container was quite embrittled and showed significant damage, though it was not broken. The FEBIAD grid looked fine, despite the large anode current draw (indicating that current was flowing elsewhere). The gas line looked fine though the gas injection to this target looked as if it wasn't connected. But there was a plug on the unused line, so it must have been correct. No other obvious signs that would explain the poor performance. Pictures are can be found here https://documents.triumf.ca/docushare/dsweb/View/Collection-37913

Looked at SiC#45 in the hot cell. Looked very clean, no deposits on the heat shield or other components. Target also looked very straight. Only issue we saw was sparking marks on the EE side that faces the ground electrode, in several spots. This target had a bout of sparking when ramping up the protons, so I guess that was the location. Will look for matching spark marks on the ground electrode next time TM4 is in the hot cell. Forgot to take picutres...

PIE on TiC#6 was done in the NHC. The target assembly looked brand new when we took it out, no heating spots or deposits (pictures on docushare). Online we measured a 400kOhm connection between TGHT-TBHT-COIL to HV common (TGHT and COIL shorted in TM2 service tray) and anode was unstable above ~300V. After the module move and before the target removal David found TGHT-TBHT-COIL was a dead short to HV common and the anode was at 39kOhms to HV common.

1. Wanted to see if the TGHT-TBHT-COIL short was on the COIL. Measured with ohmmeter the COIL lines to HS and got about 700Ohms. But with the target lying down the COIL lines drop and touch the heat shield so to do this measurement Chad had to lift the lines, which could cause them to touch something else. So not sure how reliable this measurement was. It jumped around a lot, saw 0L occasionally.

2. Measured both target legs and tantalum gas line to HS, got about 5kOhms repeatedly using the ohmmeter.

3. Meggered across the gas line insulator to the gas line going into the target and got open circuit at 250V, so the gas line wasn't coated.

4. Target container looked vaguely red in the areas where the target material was, otherwise still no deposits on the inside. The container was not overly crystalized as seen on the legs and end caps.

5. Some sparking marks on the EE and some slightly asymmetric deposits

Continuing Aug. 22 2023:

6. Found what looks like a hole in the insulation on the COIL, maybe the cause of our short? Found a corresponding line mark on the copper plates that hold the coil down but not sure if it is related. Pics on docushare.

7. Grid looks perfectly new. Where was that current going?

8. Copper plates that hold coil were loose and bolts that hold coil in just fell out as well.

9. What can be seen of the anode wire looks good.

Pumped down the gas lines in ITE on Tuesday Nov. 28 2023 and opened the CF4 bottle valves. Missed the small section between the regulator and SV2 though, so will probably redo it.

Pumped down again on Dec. 12 2023. Was taking a long time so might not be as good as earlier. Maybe there is a leak in there...

Tuesday, January 02, 2024, 15:58 :

Started HV ramp up on TM3 service tray about 3pm today. Had to bypass TCS:BIAS interlocks to make it work - there are some flow restrictions on the water lines that are causing the flow interlock to periodically trigger. The water flow it not very important for this test as long as the water is moving. Turned out that it was also conveninent to have all the interlocks bypassed because sparks tripped off the IG1 gauge and this did not cause the HV bias supply to turn off. Should use same setup next time.

Some sparking around 35-44kV, tripped off turbos/IG1/IG1S 3 times and so decided to ramp down.

Tried again while standing outside TH door to hear if sparking is outside the vacuum. Went to 48kV very slowy with multiple trips of the turbos, mostly TP2. Lots of sparks but easy to ramp back up to previous high point, so likely just a painful conditioning process. Current draw 167uA at 48kV. No sparks audible outside door, and vacuum response looks too large to be electrical noise.

Continued on Wednesday January 03 - went up to 53kV in the same way as before, lots of TP2 trips making its current a bit unstable... data was dumped as TM3_servicetray_conditioning_jan32023. Tried limiting the current on the power supply to very near the drain current to see if that reduces the number of trips. Didn't notice a significant effect.

Thursday, January 04, 2024, 09:57 :

Restarting conditioning, starting at 180uA at 53kV. Got up to around 60kV with many trips of the turbos, and one restart of the IG1 controller needed. Added a grounding hook to the vacuum controller rack to see if that helps reduce the trips. (doesnt appear to have).

Friday, January 05, 2024, 09:51 :

Continuing conditioning. Starting by sitting at 50kV for 10mins. No sparks observed during this time.

Even though we got to 60kV last time, onset of conditioning started at 58.6kV with a trip at 58.9kV... got up to 70kV.

Got to 72kV on Monday January 08 2024.

On Monday Jan. 15, tried to improve the grounding on the vacuum services rack to reduce sparking through the turbo controllers. Added two welding cables from the vacuum rack ground busbar to the copper grounding strip on the ground. If the spark is travelling from the faraday cage to the rack through conduit or grounding cables, maybe this will help. If it is coming through the turbo cable then it likely won't. Tried to add grounding straps on the turbo controller chassis (there is a brass screw which I think is for this purpose) but couldn't get either of the screws out with any resonable force. They undo a bit then get stuck...

 Wednesday, January 17, 2024, 20:39:

David vented TM3 with N, then opened the service cap to install new insulators. We pumped back down and Alexander and I ramped back up today - instabilities started around 55kV and we were back to slow conditioning in the high 50s. Got up to a set point of 65kV. Ray did some work on the TCS bias supply and now neither the set point or readback is correct. The PSU was reading 51.5kV at set point 50kV and 60kV at set point 58kV. Will continue later...

Wednesday, January 31, 2024, 16:23

Ramped up again, some slight instabiities around 56kV but could still go up pretty quick until about 65kV. Since Ray's testing of the TCS:BIAS supply both the readback and the set point are off from the powersupply display. Ran out of time and ramped down at set point 70.5.

Tuesday, February 06, 2024, 14:47

Testing again after venting the module with air. First tiny spark at 48, then some vacuum reactions starting at 49 but still no issues. Bit more reactive around 53.

 stopped at 69.4

Wednesday, February 07, 2024, 15:28 :

Ramped easily to 60kV set point and let it sit there. Didn't spark at all then after 100mins, sparked TP2 off. Otherwise first sparking happened at set point 67.

Found at limit at around 72.5 on the PSU front output, set point 70.6. think it comes from EPICS. Will have to test the power supply.... stopped there.

Thursday, February 08, 2024, 15:03 :

Long term testing. ramped back up, first spark at set point 65.

Let it sit for 2hrs 20mins. Something like 6 sparks during this time. Current also dropped significantly from 0.32A to 0.27A.

EDIT 2024-11-25, Alexander Shkuratoff:

All the StripTool data related to the above tests are on the server as the following below. Those without "Alexander" in the name were dumped by Carla.

	TM3_servicetray_conditioning_Jan32023	2024-01-03 17:29	807K
	20240103_TM3_Service_Tray_TCS_Alexander	2024-01-03 17:31	1.2M
	20240104_TM3_Service_Tray_TCS_Alexander	2024-01-04 16:58	238K
	20240104_TM3_Service_Tray_TCS_Alexander_2.csv	2024-01-05 07:00	13M
	20240104_TM3_Service_Tray_TCS_Alexander_1sec.csv	2024-01-05 12:21	81
	20240105_TM3_Service_Tray_TCS_Alexander.csv	2024-01-05 16:51	13M
	20240105_TM3_Service_Tray_TCS_Alexander_1sec.csv	2024-01-05 16:52	2.7M
	TM3_servicetray_conditioning_Jan52023	2024-01-05 21:24	8.8M
	20240108_TM3_Service_Tray_TCS_Alexander_1sec.csv	2024-01-08 16:56	4.7M
	20240108_TM3_Service_Tray_TCS_Alexander.csv	2024-01-08 16:56	7.1M

	TM3_servicetray_conditioning_Jan172024	2024-01-17 13:29	257K
	20240117_TM3_Service_Tray_TCS_Alexander.csv	2024-01-17 20:11	4.0M

	TM3_servicetray_conditioning_Jan312024	2024-01-31 17:24	2.2M
	20240131TM3_Service_Tray_TCS_Alexander.csv	2024-01-31 17:55	4.5M
	20240206_TM3_Service_Tray_TCS_Alexander.csv	2024-02-06 17:58	2.6M
	TM3_servicetray_conditioning_Feb62024	2024-02-06 19:29	4.6M
	TM3_servicetray_conditioning_Feb82024	2024-02-08 17:55	3.1M

 And the StripTool settings:

Alexander_TCS_HV_Conditioning.stp

2024-01-17 19:57

3.7K

 To deliver stable beam to RadMol, started heating UCx#43 up again. Target was overheated during initial conditioning in Nov but then ran well for 4 weeks. During the run time, the TGHT voltage jumped around and this was reflected in the ion current, but yields seemed unaffected.

When re-heating it, the TGHT voltage was very unstable and once it got to 440A, it went into voltage limiting mode. We ramped down and it was still voltage limiting, but turning things off solved it. We ramped back up and it did about the same thing. TBHT voltage and current remained steady. Looks like a heat generated issue.

Measured the resistance in the lines from the electrical room. Resistance of the lines to ITE should be 1.19mOhms for TGHT and 1.95mOhms for TBHT. TBHT positive to negative had a resistance of 7.9mOhms from the electrical room today, and 5.7mOhms on module top when measured Nov. 8. This seems to track.

TGHT positive to negative was measured at 27.6mOhms today (and was jumping around), whereas it was 7.14m Ohms on Nov. 8.

10V @ 440A is 22.7mOhms, so it makes sense that it limited. Tried with increased voltage (up to limit of 16V) but this didn't help. Declared the target dead on Jan. 11.

David measured resistances on the module top and found (in mOhms):

A-B = 27 (TGHT)

A-C = 7.473

A-D = 6.380

B-C = 26.053

B-D = 25.569

C-D = 5.212 (TBHT)

So looks like the tube is broken on the B side.

On Jan. 9 we did the PIE on Ta#66. Opened the heat shield and removed the EE. The inside of the heat shield was very black with lots of flakes falling off. The target itself looked overheated, and the container looked pretty bad for a Ta target. The outside of the EE looked perfect but after removing it there was a lot of black stuff on the side and we saw that the Re foil was curled up. Jens said during the runtime that he saw something that looked like a wire in the beam path, so maybe this was it.

UCx#43 PIE was done in the SHC. As expected based on electrical measurements, the target container was broken on the right side and came off easily. Despite all the black stuff seen on the containment box and source tray, the inside of the heat shield was very clean. Checked if the Re foil was curled in and didn't see anything. Looked fine other than the broken container.

ITW was set up for HV testing - water is running and at a high resistivity due to injected N. Einzel lens is attached to ground. Everything else is as normal. The module plug is in so no vacuum bypasses were needed. Very easy to ramp to 60kV. Sat at 60 for about 1.5hrs with maybe 1 spark (hard to say from the strip tool data). Current draw was about 176uA. Data dumped as isac/data/ITW_HVtesting_shutdown2024

ITE was set up for HV tests - this time the module plug was not in so the BIAS and EE had to be bypassed. NOTE: ITE:BIASPLC must also be bypassed for this to work. Currently it is located on the vacuum bypasses page, but may be changed to the optics page soon.

ITE was ramped to 60kV in steps of 1kV easily. It sat at 60 for about 1.5 hours with a current draw around 173uA. It was not entirely stable though, and appears to have sparked many times. None of them seemed very large, but hard to tell.

When ramping down, current dropped quickly with the first couple steps. Data saved as isac/data/ITE_HVtesting_shutdown2024

Wires were pulled in TM3 and service cap was closed up again:

Einzel lens wire (30kV kapton), routed in the pumping duct and attached with #10 ring tongue to EZL feedthrough in service cap

4 steerer wires, routed in th pumping duct and attached with push on connectors to the 4 SHV feedthroughs on the module side panel. Coax grounding sheath is cut back by about 3cm.

Multipin cable bundle #1, routed through the left cable tube in the service tray then along the heat shield water lines (with PEEK zip ties) to feedthrough A.

Multipin cable bundle #2, routed through the right cable tube in the service tray then along the heat shield water lines (with PEEK zip ties) to feedthrough B. This bundle contains the EE wire as well, which was branched out after exiting the service tray and routed along the gas lines to the EE feedthough. It was attached with a smaller (size?) ring tongue to the EE feedthrough. Do not use this wire again, very fragile. Use the Accuglass 30kV wire instead.

The multipin wires touch the back of the water blocks in the source tray (same as in MT2/4, so far no issues with that) and they also touch the COIL lines in the service cap.

The EZL and steerer wires are routed along the floor of the service cap.

Due to the difficulty of the installation, there is no metal braid over the multipin bundles in the service cap.

Photos here : https://triumfoffice365-my.sharepoint.com/:f:/g/personal/cbabcock_triumf_ca/EslWNUST-UJKh1s8jjxWG6EBpLQEnWxgV4v-NvR4ZJjlOQ?e=Mw5vvI

ITW gas lines pumped down to 2e-3Torr on leak detector with leak rate in the low e-7 atm-cc/s. Gas bottles opened at 3psi and labels in epics changed to reflect correct setup. switch box set to cathode mode, checked that anode wire was at HV common.

note ballast is removed from ITW now to be used in ITE so that line is blanked off.