Mass separator magnet timed out. Tried reconfigure. Called Kyle from controls who power cycled the CANbus card. This resolved the issue

(From the Source tray swap out.) - AS

Tuesday, June 05, 2018, 11:25: BIAS / EZL / EE @ 52 / 24 / 2.8 kV. SPARK *knocked out optics, magnet & Turbo TMS:TP6. IV4 closed/opened/reset. TP6/optics restarted. Magnet still off.

Complete failure of the communications with the magnet.

Wednesday, June 06, 2018, 13:40: Canbus contoller was swapped by Kyle L from Controls group. The old card was dead and replaced with a new one. The magnet is back and running.

Wednesday, June 06, 2018, 13:50:

It was noted yesterday during the TM2 Source HV tests that the E-rig was not running accordingly through the beam.

Upon turning on the PS, there a 45 second or so delay before the needle drops to the right at +500V and the PS no longer responds. (similar issue in August 2016)

Wednesday, June 06, 2018, 14:20: Soft reboot on Emittance rig communication. (Reboot IOC - Kyle LeBlanc)

Wednesday, June 06, 2018, 14:30: Calibrate

test scan & park

Reference parameters. Dac in bottom right corner to control PS.

Enter: 100 on Dac -- Reads 100 - On PS

Enter: 200 on Dac -- Reads 200 - On PS

Enter: 0 on Dac -- Reads 0 - On PS

Enter: -200 on Dac -- Reads -200 - On PS

The reboot fixed the erig/scanner.

TM2 Source tray still in.

Thursday, June 07, 2018, 09:20: Roughing.

Thursday, June 07, 2018, 12:42: Turbo pumps started.

Friday, June 08, 2018, 08:00: Anode run to 15 A. Short still apparent.

Friday, June 08, 2018, 08:12: Ion gauge, Turbos off.

Friday, June 08, 2018, 08:55: Venting.

Friday, June 08, 2018, 09:45: Anode connection investigated. Short found on the bracket once the anode wire was disconnected from the back screw (Picture)

Friday, June 08, 2018, 10:00: Removed 9-pin bracket. (Top connector left connected to TS. (Picture)

New Anode 9-pin connection/wire to be installed on the bracket. Test in the cold lab.

Friday, June 08, 2018, 10:12: Roughing.

Friday, June 08, 2018, 11:14: Turbo pumps started.

Friday, June 08, 2018, 12:31: Ion gauge on.

Tuesday, June 12, 2018, 06:18: Ion gauge & Turbo pumps off.

Tuesday, June 12, 2018, 07:40: Venting.

Friday, August 10, 2018, 10:25: Remove TBHT 500 PS and replace with TBHT 600 PS.

Friday, August 10, 2018, 10:50: Switch wiring in the high voltage cage from and Anode BIAS to Cathode BIAS setting.

Friday, August 10, 2018, 11:32: TBHT limit increased from 500A to 600A by Controls group.

 Friday, August 10, 2018, 11:34: TBHT tested manually. Operation is satisfactory.

Pin configuration between top 9-pin connector on Source Tray and terminal in high voltage rack. Measured 2018-08-21 by Aaron and Anders

Friday, August 31, 2018, 10:10: The RGA will not re-connect to the computer. Manually turned it off while under vacuum to protect the filament before shutting down the turbo pumps.

Friday, August 31, 2018, 10:15: Ion gauge & turbo pumps off.

Wednesday, September 05, 2018, 10:00: Communications chip replaced by RH (Travis C.)

Wednesday, September 05, 2018, 10:40: Computer restarted to reconnect/reboot.

Wednesday, September 05, 2018, 11:05: RGA started

Monday, October 15, 2018, 13:00-14:00 Anode / Cathode BIAS PS polarity reversed by John Chow.

PS: Negative

Sometime around 00:30 Sunday, October 28, 2018, the ISAC test stand lost communication to the PLC and shut down abruptly.

Monday, October 29, 2018, 09:00: PLC communications issues.

Rod N. was contacted this morning& the issues fixed by controls group.

1) When opening FC0 a spike occurs in the vacuum on IG6 & IG1 (though not as bad). This spike trips off the optics, and if left unchecked, will trip off the TGHT / TBHT / Sources.

2) RGA is also unresponsive

3) Leak rates also noted in the Ar gas bottle for the FEBIAD source. Previous trips had occurred in the most recent FEBIAD tests.

Optics tripped during FC0 removal. Vacuum spikes also detected on IG1 & IG6. No response on RGA.

Monday, November 19, 2018, 11:17: RGA providing no data. Stop, restart, multiple times. 

Monday, November 19, 2018, 11:25: RGA shut down. No response from and attempt to restart. 

Monday, November 19, 2018, 11:30: FC0 opened. Optics tripped. Vacuum increased on IG6 up to 7.9e-06T & IG1 increased to 4.4e-06T. Shut again to avoid tripping the entire system as the vacuum climbed.

Monday, November 19, 2018, 11:34: Auto-cooling @ 2A per min. Ramp down & trouble shoot after.

Monday, November 19, 2018, 14:11: Tap down HV on sources to 0V. Close & reset IV4. After closing IV4, spike in Vacuum on IG6 noted. Drop in vacuum on IG1 noted.

Monday, November 19, 2018, 14:33: FC0 opened/closed. Vacuum Spike on IG6 noted from 1.5e-06T to 1.02e-05T and back down to 1.5e-06T. IG1 unaffected with IV4 closed.

Source gas lines remain off. Setup for LD on FC0.

Tuesday, November 20, 2018, 08:30: Pump switched on. Convectron gauge @ 18.6 mTorr, spiked to 125 mTorr after opening the valve to the pump. IG1 @ 3.0e-07T.

Allow time for the small pump to pump lines down before re-opening.

Tuesday, November 20, 2018, 08:38: Small vacuum spike noticed on IG1 up to 4.0e-07T after reopening line to the small pump. Convectron gauge spiked to 1.75T.

Tuesday, November 20, 2018, 09:05: Opened valve to the small pump. Small vacuum spike noticed on IG1 up to 5.77e-07T after reopening line to the small pump. Convectron gauge spiked to 3.55 Torr from 125 mTorr.

The small leak in SV5 allows the lines to 'pump down into the mTorr range, however upon opening the line to the pump, the line pressure increases.

Tuesday, November 20, 2018, 10:44: Opened valve to the small pump. Small vacuum spike noticed on IG1 up to 5.02e-07T after reopening line to the small pump. Convectron gauge spiked to 3.25 Torr from 18.6 mTorr.

Pump ineffective at pumping out the gas lines. Replace

Tuesday, November 20, 2018, 10:48: Small pump shut off. Move forward with LD of FC0

Tuesday, November 20, 2018, 13:21 : Valve opened w/ LD connected. Base:

IG6: 1.91e-06T

Leak rate: 7.4e-09 atm-cc/sec.

Apply He around FC0 (cup in):

IG6: 1.91e-06T

Leak rate: 5.5e-05 atm-cc/sec. Wait for leak rate stablization.

Tuesday, November 20, 2018, 13:48:

Apply He around FC0 (cup out):

IG6: 2.75e-05T

Leak rate: 9.9e-05 atm-cc/sec. Wait for leak rate stabilization. Bellows specifically will max out the LD.

-Leak confirmed on the bellows of FC0 as suspected. Submitting a fault report.

-RGA scans checked and taken. No response/noted 'noise'. No error messages displayed, however we will check/swap out the filament for a new one to confirm.

Monday, November 26, 2018, 08:10: Turbos & ion gauges off.

Monday, November 26, 2018, 09:32: Quick vent shot & hold.

RGA Ionizer & Filament replaced as per SRS RGA manual.

The ionizer was found to be dirty and the previous RGA filament was broken. Initially the  old filament appeared to have warped and looked stuck to the anode grid/cage.

Both were replaced with clean parts. the support rods were also cleaned with acetone & methanol.

Parts used from the Replacement Ionizer Kit - SRS# O100RI

Repeller Cage - SRS Part# 7-00592

Anode Grid - SRS Part# 7-00591

Focus & Plate - SRS Part# 7-00590

ThO2/Ir Filament - SRS Part# 7-00611-720

Wednesday, December 05, 2018, 10:16: Roughing.

Probe bake out & RGA tuning required.

Wednesday, December 05, 2018, 11:08: Turbo pump on.

Wednesday, December 05, 2018, 11:13: Ion gauge on.

Wednesday, December 05, 2018, 13:38: TSG:BV1A Randomly shut off? TP1A tripped. Restarted.

Wednesday, December 05, 2018, 13:40: RGA switched on including the filament.

Thursday, December 06, 2018, 10:17: Vacuum @ 5.68e-07T on IG1 before Degas. Begin degas cycles 200 sec each, 1 min apart, 3 times.

Thursday, December 06, 2018, 10:30: Degas complete. Minimal change in IG1 Vac @ 5.70e-07T.

Thursday, December 06, 2018, 11:02: Turbo pumps (TP6 & TP2) off on beamline side. IG6 off.

Thursday, December 06, 2018, 14:06: Turbo pumps restarted. FC0 bellows replacement to occur Monday.

Monday, December 10, 2018, 11:00: Vented on beam side of IV4. Turbo pumps & backing valves closed preemptively on chamber side due to shared airlines.

Pulled FC0, disassemble, check, clean, remove / install bellows, Re-align, assemble.

Monday, December 10, 2018, 12:52: Re-installed repaired FC0. Roughing beam side. Chamber side turbos back on.

Monday, December 10, 2018, 13:48: Turbo pumps on.

Monday, December 10, 2018, 13:51: Ion gauge on. IG6 @ 5.73e-06T and still dropping.

Tuesday, December 11, 2018, 09:26: IG6 Vacuum @ 3.56e-07T. Connected cables.

Tuesday, December 11, 2018, 09:27: Drive out FC0. Vacuum @ 3.75e-07T. Ok. Setup LD & drive FC0 back in.

Tuesday, December 11, 2018, 09:32:  LD baseline before opening valve to beam line turbo & RP pumps: below 0.0e-09 T-l/sec

Tuesday, December 11, 2018, 09:36: Full open base, FC0 in:

Vacuum @ 3.56e-07T.

LD:  0.0e-09 atm-cc/sec

Spray w/ He:

No change.

Tuesday, December 11, 2018, 09:42: FC0 out:

Vacuum @ 3.60e-07T.

LD: 0.0e-09 atm-cc/sec.

Spray w/ He:

No Change. Sprayed all flanges and seals all around FC0 and it's connection to the beam line. OK.

Tuesday, December 11, 2018, 09:46: Closed FC0 & line to the turbo & RP pumps. LD off.

FC0 Bellows replacement a success.

More pictures uploaded to Docushare.

 Home  --->  ISAC Test Stand    Test Stand Refurbishing    FC0 Bellows Replacement Listing

Thursday, March 07, 2019, 07:48: IV4 reset, BV6 & TMS TP6 reset.

Turbo pump restarted.

Auto cooling @ 2A per min.

Thursday, March 07, 2019, 08:40: Speed up Auto-cooling to 4A per min.

Thursday, March 07, 2019, 11:06: Power cycled the control boxes for the turbo pumps one by one. Operation in back to normal.

Thursday, March 07, 2019, 11:09: Switched to filament 2 on IG6. Reset.

Sparks from BIAS to EZL noticed at 1 kV. A wire was moved away from insulator 'O'

Sparks from BIAS to EZL @ 3kV

Friday, March 08, 2019, 11:00: Power supplies checked with John Chow.

BIAS (V) ------------------- EZL (V)

Set@-Read-Actual-----Set@-Read-Actual

1000- 982- 816---------1000- 987- 996

1500-1480-1300-------1500-1475-1530

2000-1993-1600-------2000-1993-1997

2500-2491-2070-------10000-9993-9964

3000-2989-2500

3500-3487-2900

4000-3985-3360

5000-4982-4145

Note* Lower reading on BIAS. Check the scaling with controls group.

Friday, March 08, 2019, 11:14: RGA, Ion gauge, Turbo pumps off.

Friday, March 08, 2019, 13:01: Venting for visual inspection.

Visual & Multimeter checks appear ok.

Friday, March 08, 2019, 14:37: Roughing.

Friday, March 08, 2019, 15:00: Turbos on.

Monday, March 11, 2019, 09:00: Switch everything on & slowly increase EZL & BIAS together

Monday, March 11, 2019, 09:30: Spark at approx 12-13 kV on BIAS. TGHT to 600A? Readback.

TGHT PS readback issues. John C contacted.

Power down.

Prep for TGHT PS removal. Possible capacitive effect? Test TS w/o TGHT PS after.

Monday, March 11, 2019, 13:55: New TGHT PS installed.

Tuesday, March 12, 2019, 08:37: Power / sources on

Tuesday, March 12, 2019, 08:50: BIAS to over 3kV. EZL set to 1 kV, but charged to over 5.5 kV on its own.

Tuesday, March 12, 2019, 09:30: Electrical check in the HV cage revealed a light oxide coating on the EZL post 'O'. This appears to have insulated the nut & feed through. Will require venting & removal to clean.

Ion gauge, turbo pumps off.

Tuesday, March 12, 2019, 11:10: Venting.

Wednesday, March 13, 2019, 08:30: Roughing. The brass nut & spark cap were polished, cleaned and re-installed. Other connections & wires cleaned.

Wednesday, March 13, 2019, 09:10: Turbo pumps started. Wiring up top re-connected.

Update* EZL PS removed. Tested by John C & Tomislav. The HV enable circuit wouldn't activate. Upon further investigating into the control and voltage regulating circuits of the PS, they found that the control ICs are 40% under powered, only getting 10V instead of 15V from its voltage regulator.

EZL PS to be replaced.

Wednesday, March 20, 2019, 07:03: Turbo pumps off

Wednesday, March 20, 2019, 08:28: Venting

EZL checked. OK @ 5kV

BIAS PS issues.

2.3kV on screen. Read back on PS 3kV. Direct measurement reads 5.8 kV

Readback does not increase with the controls. Stops at 2.4kV & the current climbs very quickly.

Tuesday, March 26, 2019, 11:00: EZL PS routed to TS BIAS, but still controlled via EZL controls. EZL line grounded.

Brought up to 20kV, no sparks. 

New EZL PS confirmed ok. BIAS PS has issues.

Wednesday, March 27, 2019, 10:25: BIAS @ 1 kV

952V - 548uA on controls readback - Scaling req'd

964V  direct from the PS - 25 uA

Wednesday, March 27, 2019, 10:28:

2.75 kV

2725V - 1515uA on controls readback

2696V  direct from the PS - 65.8 uA - Scaling req'd

Wednesday, March 27, 2019, 10:31:

5.5kV

5465V - 3022uA on controls readback

5.478kV  direct from the PS - 137.5uA - Scaling req'd

Wednesday, March 27, 2019, 11:00: PS runs ok. Longer connection cable req'd to link to TS source. Contact controls over scaling.

Wednesday, March 27, 2019, 14:25: Sources on. BIAS connected.

Wednesday, March 27, 2019, 14:34: BIAS / EZL @ 5.5 / 1 kV. No spark.

Wednesday, March 27, 2019, 14:51: BIAS / EZL @ 10 / 5 kV. No spark.

Wednesday, March 27, 2019, 14:57: BIAS / EZL @ 20 / 10 kV. No spark.

Scaling done IOC Restarted.

Thursday, April 04, 2019, 10:33: BIAS / EZL @ 20 / 10 kV. No spark.

Thursday, April 04, 2019, 10:35: BIAS / EZL / EE @ 30 / 13 / -1.5 kV. Anode at 200 V. No spark.

Thursday, April 04, 2019, 10:49: Sources brought to 0. No spark & scaling appears complete.

Monday, June 03, 2019, 10:36: BIAS Near 55 kV, brought back down to 50 kV. Multiple sparks. Optics tripped, wire scanners & FC0 began running independently. Anode current dropped from 7.328 mA to 6.105 mA. Knocked out the TGHT

The target heater dropped from 580A to 3.1 A. Attempted reset. Non-responsive.

Monday, June 03, 2019, 10:46: TGHT Still non-responsive. Bump down HV to open the cage & cycle the PS.

PS model: Xantrex XPR 10-600

Message lights on the front of the power supply:

OTP - INT - OVP - SNS Triggered.

Thursday, June 13, 2019, 14:05: ELOG 732 - PS replaced.

Monday, June 24, 2019, 15:25: Current quadrupole polarities:

Monday, June 24, 2019, 15:35 : The polarity on TMS:Q3 is reversed. The top and bottom poles should be at a positive voltage similar to TMS:Q4, however a negative tagged cable is connected to it.

Monday, June 24, 2019, 15:39 : Swapping the cables on TMS:Q3.

Tuesday, June 25, 2019, 16:15: While inspecting the 9-pin connection with Ferran, it was noticed how two braided cables were burnt out pretty bad to the point they were separated in half (see attached pictures).  Additionally the plastic insulator for the anode wire was melted.

Tuesday, June 25, 2019, 16:30: Braided cables need to be changed and anode wire redone. 

Tuesday, June 25, 2019, 17:16: Wondering if this happened during TM4 source tray conditioning given the following entries in the elog (entry 725):

Monday, June 03, 2019, 09:51: BIAS down to 50 kV. Burnt plastic smell lingering. Wait* 

Tuesday, June 04, 2019, 10:29: Note* Minute fluctuations of Anode current have ended after the spark. Also slight 'Burnt' smell noticed around the test stand area.

The wiring on the top of the test stand was burned, see entry 737.

Probable cause of this is as follows:

Coil lines are connected by a wire to HV common, the 9-pin feedthrough etc to maintain same voltage on all the insulator tops. This is also directly connected to the extraction electrode spark gap. When this sparks, it momentarily changes the potential in that line (normally HV common) which then can make this path the best route for current from the coil to return to ground.

See attached diagram.

Best solution to this problem is to disconnect the high current section from the low current section, and attach each separately to HV common.

Other possible improvements to wiring:

1. Remove jumpers between positive coil lines (and neg. coil lines) since these feethroughs are physically connected by the copper pipe.

2. Add a resistor to the jumper between coil - and TBHT - since this will eliminate all possibility of current flowing between the two.

3. Add a resistor in line where the high current section is connected to HV common to avoid possibility of current going through jumpers.

See elog 737 

Thursday, June 27, 2019, 09:45: The cables showing burnt marks were changed with new ones on Wednesday 26th. The grounding cables were insulated with yellow heatshrink. The anode wire connection was redone.

Thursday, July 04, 2019, 15:00: TBHT insulator was swapped out on the source tray side. 

The old one is very chipped.