The five spent target F308  flasks where gone through in accordance with Nordion guidelines, the sixth spare is remains in the Hot Cell area. All flasks where in reasonable shape with two needing weld repairs on the pallets. I replaced all the 3/4" pallet to flask fasteners with new Grade 8 bolts nut and washer combo. Flasks have been moved into the Hot Cell area awaiting movement into the Target Hall.

A  hydraulic leak was found on the manipulator  install and repair cart, upon further investigation the cause was a faulty cylinder. Once the cylinder was removed and disassembled the inner bore was found to be scored beyond repair. Another point of interest was that the rod was painted to the point that when retracted it compromised the rod seal causing the cylinder to leak. attached are photos outlining the issues. new cylinders are being sourced and will be installed upon arrival. We have found that there is interaction between the tops of the cylinders and the bed of the cart when fully retracted and compressed, this might require some modifications to ensure proper operation in the future.

Saturday, April 27, 2024, 09:35 :

Started ramping up HV, EE and EL together. First small spark at 20.5kV on bias, no reaction in other devices.

Note both BIAS and EL are off in their readbacks. See table below

Saturday, April 27, 2024, 10:23 :

Spark on HV only at 25.5, couple sparks from here to 33, then at 33 starts real conditioning.

EL is reacting to some of the sparks, not sure if this is just electrical noise or not.

Saturday, April 27, 2024, 18:14 :

lots of trips above 60kV on the set pt. tripping off turbos and gauges. IG2 needed the controller reset.

RESULT: Got to 64.5 (actual) 62 (set point) with quite a bit of sparking over 60kV. Will go the rest of the way to 73 in the next days. Data dumped as isac/data/TM3_HVtests_Apr272024.dat

Sunday, April 28, 2024, 11:23 :

Ramped back up to 60kV reasonably easily, couple sparks and turbo trips.

Went up to 64kV (set point) with a lot of difficulty. Data dumped as isac/data/TM3_HVtests_Apr282024.dat

Monday, April 29, 2024, 14:58 :

Ramped up to set pt 62 easily, went down to 500V then 100V steps after that.

Went to set pt of 67 and left it there for ~40mins. Not perfectly stable but not too bad.

Tuesday, April 30, 2024, 17:12 :

After rebooting the TCS PLC to fix all the non-responsive devices, everything is working fine again.

Ramped easily to 60kV in 1kV steps.

Still got to 67 without too much trouble. Behaving pretty well...

At 70.2 (set pt, 71.3 readback) the water and vacuum gauge both tripped off and cannot be restarted. I am not sure what the real power supply reading is here but in the past it has been 70.5 set pt = 73 real value, which is as high as the power supply goes.

Stopping here since I'm not sure the water is circulating... the off pump says it is and the on pump says its not... left it at 70.2 for 40mins, still sparking going on (20:28 to 21:10)

Data dumped as isac/data/TM3_HVtests_Apr302024.dat

Sunday, May 05, 2024, 10:50 :

Containment box and target installed.

HV conditioning starting at 47kV.

Spark at 58kV knocked out IG1. Continuing with IG2 and IG1S.

Went up to set point of 63, readback 64. Will continue on Monday.

Data dumped: isac/data/TM3_HVtests_May52024.dat

Monday, May 06, 2024, 09:37 :

Ramping system back up.

First sparks at 50kv while raising the EL voltage (not sure if they are related).

AS: Voltage quickly up to 63 kV from 49 kV. Very smooth, no sparks or instabilities. Ramped at around 250 V steps every 10-20 seconds. First signs of instability at ~62.75 kV and definitely at 63 kV. I guess this is where it was conditioned to last. Let sit at 63 kV for a while while doing something else. It sparked once and turned off a turbo after half an hour. Starting ramping again.

AS: Spark at 65.75 kV killed IG1. Carla manually reset at 69 kV. Still working. A spark at 70.25 kV took out IG1 again.

AS: Maximum voltage reached at about 16:45. Setpoint 70.5 kV, readback 71.7 kV, PS itself ~73 kV. Carla reset IG1.

AS: 18:30, spent about 1.5 hours cumulatively at this maximum voltage. During this time it sparked about 15 times. Not long enough to tell if the spark frequency is decreasing. In the middle of this, Carla did something accidentally and caused the voltage to drop a bunch, so I manually set it to 0 V. She then reset TP1S, which was not staying on. Also, the EE and EL stayed at their respective voltages during this downtime (2 kV and 28.7 kV). I ramped the main BIAS voltage back up to the maximum voltage within 5 minutes and it was smooth all the way but sparked again at the maximum voltage. It seems conditioned up to or near the max voltage. Ramped down and turned off BIAS, EL, and EE.

AS: Dumped data to isac/data/TM3_HVtests_May62024_Alexander.dat

Tuesday, May 07, 2024, 10:02 :

Tested the EE - ramped up easily to 7kV, but then had issues maintaining the voltage. No vacuum response. Ramped down since we never use it that high anyway, though cable should be good for 10kV. More investigations later.

EDIT 2024-11-25, Alexander Shkuratoff:

Summary of StripTool data taken during this round of tests:

	TM3_HVtests_Apr272024.dat	2024-04-27 18:56	12M
	TM3_HVtests_Apr282024.dat	2024-04-28 13:16	1.4M
	TM3_HVtests_Apr302024.dat	2024-04-30 21:11	9.0M
	TM3_HVtests_May52024.dat	2024-05-05 13:06	5.0M
	TM3_HVtests_May62024_Alexander.dat	2024-05-06 18:45	9.5M

TM4 in conditioning station with no source tray, extraction electrode set to HV bias. Brought up to 30kV over the weekend by ISAC ops with no issues.

BIAS=30kV

EL=15kV

IG1=9.4e-7T

Monday, June 10, 2019, 08:21 : At 32.4kV on bias, EL at 15kV, bias sparked to EL. Every bump of 100V on bias is accompanied by a small vacuum jump and often by a small current increase from ~60uA to 120uA.

* Conditioning station read back and set values are off by 2kV but archive values show this has always been the case. Which to trust? *

Monday, June 10, 2019, 08:54 : Spark while sitting at 38kV on bias, 19kV on EL. Seems to be a spark to EL.  Sitting at 38kV for ~10mins

Monday, June 10, 2019, 09:32 : Sat at 40kV for ~10mins or so, bumping up again.

Monday, June 10, 2019, 09:38 : Spark to EL while sitting at BIAS / EL = 42kV / 20.5kV

Monday, June 10, 2019, 13:57 : Ramped to BIAS / EL = 43 / 21.2. No problems other than what seems to be the usual conditioning station stuff.

 Monday, June 10, 2019, 14:17  : Couple sparks on the way to 48kV, seems to be to the EL.

Monday, June 10, 2019, 14:36 : Up to BIAS/EL = 51/25.5 with no big problems. Will stay here for a couple mins

Ramped down the bias so David could remove the HV cover and lock out the target hall.

Ramped back up to BIAS / EL = 50kV / 26kV quickly with no sparks.

Monday, June 10, 2019, 15:35 : BIAS / EL = 52kV / 26kV

Monday, June 10, 2019, 15:45 : BIAS / EL = 54kV / 28.5kV. It seems that keeping the EL at 1/2 the bias voltage is right on the edge of what is acceptable - the HV seems to be sparking to the EL instead of to ground (either that or the EM interference from the sparks it producing a reaction in the EL...) I will run the EL higher and see if that helps.

Monday, June 10, 2019, 15:57 : Sat at 54kV for > 10mins, will continue up

Monday, June 10, 2019, 16:11 : At BIAS / EL = 55.8kV / 29kV got some sparks, some to EL and some to ground. Back at 55kV.

Monday, June 10, 2019, 16:24 : Sitting at 56kV for a couple mins

Monday, June 10, 2019, 16:37 : Sitting at 56.3kV bias, a spark tripped off the IG1S, which tripped the turbo and shut down all the HV. Back to zero!

Monday, June 10, 2019, 16:59 : TM4 back up to BIAS / EL = 56.1kV / 29.5kV

Monday, June 10, 2019, 17:14 : After 57kV on the bias, things seem to be getting sketchy. The EL voltage and current are a bit jumpy and stepping that voltage up seems to produce a slight spike in the bias current.

Monday, June 10, 2019, 17:31 : At BIAS / EL = 58 / 30.3, looks ok.

Monday, June 10, 2019, 18:11 : Sitting at 59.5kV, could probably push further but will sit here for ~30mins and make sure it is relatively stable. Note 59.5kV set point is really 56.9kV on readout.

Monday, June 10, 2019, 18:18 : some current inconsistencies (not really sparks probably corona discharges...) so went down to 59kV

Monday, June 10, 2019, 18:23 : still getting some sparks at 58kV, going down to 57.5.

Monday, June 10, 2019, 18:37 : went back up to 58, spent 10 mins there but was getting these corona-discharge-type-sparks where the vacuum responds and current is drawn but voltage is maintained. Went down to 57kV, still getting them....

Monday, June 10, 2019, 18:40 : Going down to 56 to see if they stop....

Monday, June 10, 2019, 18:42 : Trying 55 kV now... see attached screenshot for example of sparks

Monday, June 10, 2019, 18:48 : No luck at 55 or 54kV. Going down to BIAS / EL = 50 / 27 for a while to see if this resolves

Monday, June 10, 2019, 19:20 : 30 mins at 50kV, a couple little sparks, but no big deal. Ramping up to 55 and let's see if it is ok.

Monday, June 10, 2019, 19:30 : 55 still not working, going down to 54

Monday, June 10, 2019, 20:16 : Held bias at 54kV for about an hour - dropped it down once to alleviate some sparking. Leaving it at 50kV overnight, ops have instructions to keep an eye on it and shut it off if anything seems strange.

The conditioning station was ramped down overnight and David ramped it back up this morning. Currently trying to get it to sit stable at something higher than 55kV.

Wednesday, June 12, 2019, 08:11 : Sitting at BIAS / EL = 53.9 / 28, 2 sparks to ground, one turned off TP2

Wednesday, June 12, 2019, 08:53 : Been at BIAS / EL = 56 / 28 for >10mins with no real sparks. Will continue up.

Wednesday, June 12, 2019, 08:54 : Spark to EL at 56.1

Wednesday, June 12, 2019, 09:34: Sitting at 55kV for 40 mins. 2 sparks to ground, 1 spark to the EL.

Wednesday, June 12, 2019, 09:49 : Leaving it at 56.6 for a while

Wednesday, June 12, 2019, 09:56 : Either there was a spark to IG1S or there is a vacuum limit that does not accept a 1.2e-6T spike. Everything turned off, but everything seems to still be working.

Ramping up TM4 with new source tray in. Current condition: new source tray but no water running in lines B, D or window. Multipin on the source tray grounded but no lines running into the service tray. Vacuum at 1e-6T.

Small sparks at each BIAS voltage increase starting at 30kV. First real spark around 38kV.

Going up in more than 100V steps causes spikes that take out TP2.

Monday, July 08, 2019, 14:54 : TP2 sparked off at BIAS/EZL = 44kV/25.1kV after 10mins at this point... will leave it here for a couple hours

Monday, July 08, 2019, 16:44 : 6 sparks in last 2hrs, some very strange vacuum response in which the pressure rose steadily and stayed high for 30mins before coming back to normal.

Monday, July 08, 2019, 17:25 : After 30mins at 45kV, started sparking, so will ramp down to 40kV and leave it overnight

Left TM4 at 40kV overnight with no previous heating. There was only one real spark which turned off TP2 but ops turned it back on. Ramping up again.

Tuesday, July 09, 2019, 09:13 : At 45kV the module is sparking on the top pretty regularly. Turned down and David will remove cover and lock out target hall.

Tuesday, July 09, 2019, 16:30 : Went up to 45kV pretty easily now.

Tuesday, July 09, 2019, 16:32 : Everything tripped at 46.5kV. Looks to be a spark outside the vacuum. TM4 sparking much more outside the vacuum than last checks done without the source tray...

Tuesday, July 09, 2019, 16:43 : Everything easy up to 46.8kV, then sparking a bit. Most of the sparks are outside the vacuum. Sitting at BIAS / EZL = 47.1kV / 28kV for a couple mins before ramping down.

Tuesday, July 09, 2019, 16:57 : 10 mins at 47.1kV with not problems. Continuing up still looks like it is really sparking a lot outside the vacuum.

David raised the bias to 49.5kV. Module is still sparking mainly on top. Humidity is super high.

Wednesday, July 10, 2019, 09:50 : sitting at 49.7kV for a coupe mins, big spark in vacuum took everything down.

Wednesday, July 10, 2019, 10:16 : Bias at 50kV drawing 0.1mA but looking pretty jumpy. Left it here for ~5 mins then had to turn down bias for TH work.

Wednesday, July 10, 2019, 13:33 : Back at 50kV on bias with no problems.

Wednesday, July 10, 2019, 13:46 : 10 mins at 50kV, will try to go a bit higher

Wednesday, July 10, 2019, 14:06 : Everything sparked off while sitting at 50.6kV. This seems a bit futile....

Wednesday, July 10, 2019, 14:15 : Back at 50.1kV with no problems

 Wednesday, July 10, 2019, 14:33 : Up to 51kV

Wednesday, July 10, 2019, 15:20 : Difficult to go above 51.4. Turning up EZL since there were a couple sparks to it. -> Putting this up causes significant sparks between HV bias and EZL in vacuum at 5kV on the EZL, 51.1kV on bias. Leaving EZL @ 5kV

Wednesday, July 10, 2019, 16:04 : Around 52.2 sparking on the top of the module

Wednesday, July 10, 2019, 16:36 : 53kV

Wednesday, July 10, 2019, 17:16 : up to 54kV without much trouble

Wednesday, July 10, 2019, 17:53 : Sitting at 55kV for about 4mins when a spark tripped off everything. Will try to bring it back to sit stable at 55kV tomorrow

Thursday, July 11, 2019, 10:28 : TM4 up to 54.9kV without too many problems, EZL at 25kV. Will sit here for a little while

Thursday, July 11, 2019, 10:48 : 20mins at 54.9kV, going to 55kV

Thursday, July 11, 2019, 11:14 : 25 mins at 55kV and everything sparked off

Thursday, July 11, 2019, 11:26 : Back up to BIAS / EZL = 54.8kV / 26kV. Checked that the TGHT and TBHT power supplies have not been killed by sparks, everything looks fine.

Thursday, July 11, 2019, 12:56 : Sat at 49.6kV for 1.5hrs with one spark.

Thursday, July 11, 2019, 14:02 : Been sitting at 55kV for 2 hrs now with one spark.

Friday, July 12, 2019, 08:09 : Back up to 52kV easily, then had to reduce step size to 100V to keep going

Friday, July 12, 2019, 08:47 : 56kV

everything sparked off at 56.5 or so

Original Problem:

Leaks on the B and D copper blocks, probably due to corrosion between the blocks and the inserts. Final solution, after many tries, was to make an indium sheet the size of the copper blocks and use that to seal the entire surface.

So far offline tests seem to show these seals working, though long term tests are needed to say for sure.

New Problem:

TM4 was in the conditioning station, running water and under vacuum. 50A on the target and tube heaters from Friday evening to Saturday afternoon.

Sunday morning 9:30am target heating increased to 200A, 100A on the tube heater, ramp finishing around 1:30pm.

At around 3pm, the vacuum stops dropping and begins to climb very slowly, continuing until Monday morning (see first attachment).

Tests on the individual water lines show an immediate vacuum response only to the EE lines, the other lines show no response (note the window line is blanked off since the containment box is not on).

With water in the EE lines, the vacuum sits around 1e-6 near the target.

Testing:

Ramped target to 400A and tube to 200A. Sitting there for several hours shows no change in the vacuum response, though there is a variation that corresponds to the chiller turning on and off (see second attachment).

Ramped target to 450A and tube stayed at 200A. Vacuum staying in the low 10^-6 range, seems to be entirely due to EE leak, other seals are holding well.

TM4 has developed 4 leaks so far:

1. Line B - seems to be solved with an indium gasket (some testing already done)
2. Line D - seems to be solved with an indium gasket (some testing already done)
3. EE - indium gasket installed, preliminary tests are good
4. Window cooling line - David found with 70 psi helium, leak detector response went from 3E-7 atm.cc/sec to 4E-4torr atm.cc/sec in 1 min and stayed there. When water was put through the circuit, there was an initial vacuum increase to about 1E-6, then a couple hours later it began to fall and has been falling for ~14hrs, now in the 7E-7 range for IG1.

Testing indium seals on lines B, D, EE. No repair has been done for window line. Ramping heaters to TGHT=450A, TBHT=200A. Everything behaving normally.

Wednesday, ramped up BIAS, EZL and EE to 36kV, 17kV, 1.5kV. Sparking at about 1 spark per 20 - 30 mins. Bias current somewhere between 66uA and 78uA, varying with the chiller ON/OFF.

Thursday morning around 2am everything tripped off. No spark visible on strip tool, seems this is from an increase in the window water leak. Vacuum recovered on its own and heaters were ramped back up.

Starting HV tests on TM2 with 50uA of protons.

Tuesday, August 06, 2019, 03:37 : Ramping up 500V every 2mins or so, couple of sparks, then consistent sparks at 34kV. No vacuum response on IG1 for any sparks, even though the current spikes from around 40uA to 1300uA. Brought it down to 28kV to stop sparking. See a response on the EZL frequently.

Tuesday, August 06, 2019, 05:24 : Going above 30kV proves very difficult, lots of up and down, frequent sparks, still no vacuum response. Still seeing an EZL response.

Tuesday, August 06, 2019, 05:58 : Up to 31.5kV on bias, 1046V on EE and 15kV on EZL.

Tuesday, August 06, 2019, 06:50 : Controls are being sticky - not responding well.

Tuesday, August 06, 2019, 07:25 : Sat at 32 kV > 20mins with 2 sparks

Tuesday, August 06, 2019, 08:03 : lots of sparking at 36kV, went back down to 35kV. Current draw is 50uA.

Tuesday, August 06, 2019, 08:10 : lots of sparking after 9 mins at 35kV, had to turn it down to 34kV

Tuesday, August 06, 2019, 08:11 : Still sparking at 34, down to 33

Tuesday, August 06, 2019, 08:31 : At 33kV, sparked twice in 20mins and was going for another when the protons went off.

With protons off, the bias current dropped quickly from 66uA to 40uA and raising the bias to 40kV was not difficult.  Later bias up to 50kV. Still no conclusion on if the sparking was inside or outside of the vacuum. No vacuum response seems to indicate outside, however the influence of protons would indicate inside. To be investigated visually when TM2 comes to the HC.

Ramped voltage on TM2 up to 30kV with no problems. Bias current is at 29uA, same as original TM2 bias current (see archiver for ITE June 15th, before protons) and similar to 31uA right after protons went off on Aug. 6.

TGHT=0A

TBHT=0A

BIAS V=30kV

BIAS current = 29uA

EZL V= 15kV

EZL current= 51uA

EE V= 1kV

EE current= 0.952mA

IG1= 8.6e-7

protons=0

This seems ok despite poor electrical checks showing 300 MOhm resistance between HV and ground. These values indicate a resistance of 1 GOhm. Let's see what happens when protons are on.

For future (optimistic) reference when testing protons:

BIAS=35kV, 35uA

EZL-17.5kV, 60uA

BIAS=38kV, 37uA

EZL=18.5kV, 63uA

Ramping up HV to ~30kV, encountered problems and had to drop back to 28kV. No response on IG1 or anything else during sparks. Lots of charging-type behaviour on the bias current. Protons at 55uA

Tuesday, September 17, 2019, 02:02 : 15 mins with only 1 spark at 30kV so increased the bias

Tuesday, September 17, 2019, 02:31 : 32kV. Current unstable

Tuesday, September 17, 2019, 02:41 : Current continues extremely unstable but not much sparking....

Tuesday, September 17, 2019, 02:53 : Proton beam trip initially settled the bias current down, but then current began climbing pretty steeply with protons still off

Tuesday, September 17, 2019, 03:28 : Protons dropped to 40uA from 55uA, Bias current is much more stable and no sparks for 25mins

Tuesday, September 17, 2019, 03:36 : Raising voltage easy to about 33.5kV then instability and sparking similar to around 31kV at 55uA p+

Tuesday, September 17, 2019, 04:45 : Lowered protons to 25uA and no sparks though the current is not stable

More things but forgot to write them down. Spreadsheet attached.

Tested RF from RF driver into new IGLIS wires at ITE. Used 5V, 1MHz (2MHz setting because of internal halving issue).

As expected there was a very slight mismatch visible (original cable ~ 68 Ohms, new cable ~ 53 Ohms).

The matching resistor will be changed and it will be retested.

Finally ITE:EL is still not working. It shows a value on the front of the PSU but doesn't draw enough current and doesn't affect the beam. We are running without it and it will hopefully be fixed on the Aug. 4 maintenance day.

These electrical issues bear some resemblance to those with Ta61, but I don't think we ever determined the cause there.

See the attached for the way ITE has been re-wired for the Ta62 run. Note also that the copper block in the schematic on TBHT negative may not actually be connected. Wiring verification on Aug. 4.

TM2 is in the TCS without containment box for HV testing following the installation of the IGLIS wiring.

Not great results so far - sat at 42.5kV for nearly 2 hours, current around 200uA and occasional sparks. I did not have time to go higher, will do this tomorrow. The sparks could be conditioning but it looks a bit suspicious to me. We will see if the ramp up tomorrow is very easy and how high we can go. I have data from HV testing of TM2 in December for comparison.

The EZL was reacting to all sparks, so I went into the target hall to see if it was sparking on the module top, but there was no sparking there.

Possible mitigations:

1. Re-do the connections on top of the service cap to remove the extra length on the wires inside the service cap and inspect the routing carefully to make sure nothing is in the wron place.
2. Inspect the wiring in the source tray to see if there is evidence of sparking in a certain area.

Second round of testing Tuesday April 5 2022:

Much better results, seems most of the sparking was conditioning. Went right up to 42.5kV with no issues. Spent quite a while getting up to 50kV, and didn't leave it there for more than a couple mins. May be running into the limitations of the HV cage at this point. Bias current was much lower, something like 140uA at 45kV.

Third round of testing Wednesday April 6 2022:

Ramped easily back to 45kV, a little more sparking above that. Sat at 47kV for ~30mins with no sparks. Won't try to go higher in case of issues with the cage. Will try to condition up to 50kV in the station. Other than the possible low megger value between B and C, all looks good.

Chad changed the EZL cable and plugs on TM2. New cable runs along the cable trays.