Andrew Zonenberg

Security and open source at the hardware/software interface. Embedded sec @ IOActive. Lead dev of ngscopeclient/libscopehal. GHz probe designer. Open source networking hardware. "So others may live"

Toots searchable on tootfinder.

Public Key

npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd

NIP-05 Address

azonenberg@ioc-exchange.mostr.pub

Profile Code

nprofile1qqsvxk504cz6sw9cvdc8sqamrl0uljanntw0q8hp48799m6ynxxwfsqpz4mhxue69uhhyetvv9ujuerfw36x7tnsw43qfqrw8c

Publishing to

relay.ditto.pub

Author Public Key

npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd

Show more details

Last Notes

2026-05-16 19:17:44 UTC

- reply

@nprofile…fqfm @nprofile…cy5p I'm hoping one day a bunch of angry nerds will design either an entire suite of new electronics/controls for a popular late model vehicle, or better yet an entirely new vehicle, that is simple, low tech, repairable, and open source to the extent practical.
Rip out the CAN bus controlled door locks and replace them with ones that use physical keys that don't need batteries.
Remove the touchscreen HVAC controls and put in physical knobs.
No fly-by-wire anywhere except where required to perform a key safety function like ABS. If you have to put software between human control inputs and steering/braking/acceleration, it should be as minimal as possible, formally verified, and multiply redundant processors, sensors, and actuators.
Publish full mechanical drawings of common wear components to make third party replacements easier and allow the vehicles to outlive the manufacturers if they later go out of business.
No ADAS features whatsoever.

2026-05-16 18:26:21 UTC

- reply

@nprofile…cy5p I am absolutely prepared to excise radio power amplifiers with a hot air rework station if it comes down to that level though.

2026-05-16 18:25:45 UTC

- reply

@nprofile…cy5p If I am ever forced to buy a 2015+ vehicle (or whenever they started putting telematics on cars) due to lack of alternatives, you can bet I am going to go bumper to bumper with a spectrum analyzer and nearfield probe set and neutralize every radio transmitter on the car by any means necessary.
Removing the entire ECU? Cutting traces to the antenna and replacing them with a 50-ohm terminator? High voltage into the antenna until it stops responding?
One way or another, it won't phone home. But the longer it takes and the more work it requires, the more pissed I'll be.

2026-05-16 18:23:46 UTC

- reply

@nprofile…cy5p I work in computer security and do a lot of work on automotive stuff.
People ask me what I think of self-driving cars.
I tell them I get to work on a bicycle. The smartest thing on it is the USB charge controller for my headlight and if it makes a funny noise or starts smoking I can yank it off the handlebars and yeet it somewhere far away from me.

2026-05-16 01:51:31 UTC

Load break cutouts on a cross connect downtown
https://files.ioc.exchange/iocexchange/media_attachments/files/116/581/723/195/724/440/original/9289c8195bbe08b8.jpg

2026-05-15 22:42:57 UTC

- reply

@nprofile…u6vt yeah a good trainer will recognize when someone is at the breaking point and pull them before they get hurt, or set a max number of reps. you're not trying to kill the team you want to build them up

2026-05-15 22:34:07 UTC

- reply

As written this probably wouldn't work but it's inspired by the military / sport training practice of doing a group run across a field or something.
The first one to the other side gets to take a shower. Everyone else does it again.
The idea isn't to punish the worse performers, it's to give more practice/exercise to those who are underperforming until everyone is able to meet the standard eventually.

2026-05-15 22:31:18 UTC

Cursed exam idea: everyone gets the same problem, first to submit a correct answer gets a passing grade.
Everyone else gets another problem.
The exam will continue until everyone passes.

2026-05-13 06:00:40 UTC

Progress! CI jobs for two consecutive commits dispatched by GitHub commit hook.
First one had some minor issues, I fixed the CI script, then pushed a new commit before the first build had finished running.
If everything works, we should get a full build of both commits soon.
I've started work on the infrastructure for artifact uploading but need another day or two to get that all wrapped up.
https://files.ioc.exchange/iocexchange/media_attachments/files/116/565/734/298/291/923/original/2091e189e76c2118.png

2026-05-13 00:26:12 UTC

- reply

@nprofile…thd8 that sounds like something a scopehal filter could do. You'd need to write a boundary scan decoder to go from the jtag scans to digital waveforms for each IO, but then you could run the existing spi decodes on top of that.
Complex chains of tunneled protocols is something we can handle with ease given the right intermediate blocks

2026-05-12 22:46:21 UTC

It's been a while since I've done web stuff. Did I screw up anything too horribly here?
https://github.com/ngscopeclient/scopehal-ci-scripts/blob/main/api/github-hook.php
This is a web hook that is only triggered by the 'push' event on ngscopeclient/scopehal-apps.
Goal is to ensure that
a) nobody but github can trigger builds (to prevent DoSing the CI platform with a ridiculous number of builds etc)
b) a compromise of GitHub's webhook infrastructure can trigger builds of real commits in the repo, but not run arbitrary shell commands on the CI runner or pull from an untrusted fork (i.e. no shell command injections etc in the branch/commit strings which will be fed to a zillion bash scripts downstream)

2026-05-12 19:32:26 UTC

- reply

@nprofile…ffys seriously, i love raw vulkan with minimal wrappers for compute acceleration... it's almost as low level as CUDA but it's not absurdly heavyweight tooling wise and is portable to anything from a Haswell-era integrated GPU to a modern nvidia monster to a macbook air

2026-05-12 19:31:13 UTC

- reply

@nprofile…ffys apt install libvulkan-dev
Problem solved?

2026-05-09 13:41:59 UTC

- reply

@nprofile…t9xu if i did somehow have ~4x the generation capacity i in fact do, I'd have a different problem though... Anything that can store tens of MWh of extra energy can let it out again if something goes wrong.
But that's a problem i would save for somebody else because I'm on a net metering plan that resets at the end of March which conveniently aligns with when solar production really picks up for the year.
So if i had a ~72 kW peak solar farm instead of the 18 I do in reality, I'd just throw my extra power at the utility to use for other customers and draw down my negative bill balance over the winter. The vast majority of our power here in WA is renewable anyway (hydro and wind for the most part)

2026-05-09 08:13:11 UTC

- reply

@nprofile…t9xu I burn all of my surplus overnight, there's nothing to save for later anyway

2026-05-08 18:24:49 UTC

398 kWh solar generation for May so far and it's not even noon on the 8th.
This is more power than I generated during all of February (393 kWh) lol. Solar season is definitely a thing.

2026-05-07 12:00:18 UTC

Hmmm... How likely are platform specific memory safety issues in ngscopeclient/libscopehal? I'm thinking "not very" since almost all of the OS-specific stuff code is in third party portability libraries.
I ask because right now we do all of our CI builds as optimized release builds with the intention of creating nightly builds other people can download and play with.
But I want to do ASAN+debug builds to catch memory safety issues during tests (there was at least one that this could have caught in the past).
The question is, do we double up *all* of the CI builds, using twice as much time/power? Or is it sufficient to do a *single* debug/sanitizer build, on one supported platform (probably one using llvmpipe to avoid monopolizing a real GPU)?

2026-05-06 21:01:54 UTC

- reply

I was hoping I'd be sub $700 for April but didn't quite generate enough juice to do it.
May started out strong with four days of >60 kWh solar generation (which counted as part of the April billing cycle) before some clouds rolled in, so we'll see... May around here tends to be the transition between the monsoon season and cloudless summer weather.
Hoping to hit below 3 MWh imported in May, so let's see how that goes.
Here's some graphs from my analysis spreadsheet showing my demand vs utility rates over the last few years.
https://files.ioc.exchange/iocexchange/media_attachments/files/116/529/636/620/185/035/original/c9af5918369a39b5.png
https://files.ioc.exchange/iocexchange/media_attachments/files/116/529/639/468/136/893/original/324b3adc555abe65.png

2026-05-06 20:56:35 UTC

Got the official power numbers from my utility for the April billing cycle.
3572 kWh imported from grid, 200 kWh overproduction exported, for net 3372 kWh consumed/billed.
That is an all-time record for lowest power consumption (at least back to January 2024 which is the oldest bill I have easy access to).
Too bad it's also an all-time *high* for price per kWh - the effective price per kWh after taxes and fees hit 21.7 cents, a 15.4% increase since the start of the year and 53.9% since January 2024.
Even so, the $731 bill for 3372 kWh is the smallest since I paid $722 for 3983 kWh back in July 2025. And that was an outlier, going further back the next time I had a bill that small was $618 in October 2024 for 3998 kWh.

2026-05-04 18:17:53 UTC

- reply

@nprofile…02n5 something something taylor swift private jet

2026-05-04 09:33:59 UTC

- reply

For comparison... my 16 GHz LeCroy oscilloscope puts out 40 Gsps * 4 channels * 8 bits of raw ADC samples, not counting the flatness corrections done in gateware/firmware.
That's 160 GB/s or 1.28 Tbps of raw samples.
That would even fit in NVLink 2.0 much less the current gen4/5 stuff.
Imagine four channels of 16 GHz bandwidth waveform data straight into a (very large) GPU nonstop... We'd have to do a hell of a lot of optimization to ngscopeclient to keep up and probably add multi-GPU support but it would be so much fun lol.

2026-05-04 09:29:03 UTC

You know what would be really nice (but nobody is ever going to build)?
Oscilloscope that replaces the ungodly slow USB3/1000baseT PC interface port with NVLink.
Forget PCIe and Thunderbolt... 900 GB/s of bandwidth straight from the ADC to my GPU? Sign me up.

2026-05-03 18:51:26 UTC

Whenever me and my wife are driving somewhere together and are backing out of a driveway, making a turn at a sketchy intersection, etc whoever is in the passenger seat always acts as a second set of eyes calling out traffic or "clear right" etc as appropriate.
From what I've seen this is common practice in military and fire department circles but pretty rare in the civilian world for some reason. Not sure why, better situational awareness definitely makes things safer.

2026-05-02 15:17:16 UTC

- reply

@nprofile…a65n it claimed submersible and i certainly got it pretty wet over the years. Pretty sure it's been submerged at least once.

2026-05-02 14:25:21 UTC

- reply

@nprofile…a65n air quality + IP rated might be harder since you'll need the sensor exposed

2026-05-02 14:03:39 UTC

- reply

@nprofile…a65n doesn't exist afaik

2026-05-02 08:57:47 UTC

I have a pair of end-of-life phones (CAT S61 and S62) with FLIR Lepton thermal cores in them.
Anybody know if there is a good way to disassemble them to salvage the thermal modules without destroying them? I don't care about the rest of the phone, it's gonna end up in ewaste.

2026-05-02 05:20:15 UTC

I just tried to ctrl-O to save a file in a GUI text editor.
I think I've been spending too much time in nano lately

2026-05-01 16:41:37 UTC

- reply

@nprofile…3tsk lol moss is native to this area the grass is probably what doesn't belong

2026-05-01 16:36:14 UTC

- reply

@nprofile…3tsk the yard is like half moss already I just need to get rid of the grass and buttercups lol

2026-05-01 16:35:47 UTC

- reply

@nprofile…3tsk i would like to replace our grass with moss... Never needs mowing, nice and soft

2026-05-01 14:48:34 UTC

Having a bit of fun with my 100G LAN, new GPU, and a pair of ThunderScopes.
Here's not one but *two* 1 Gsps live waveform streams being FFT'd simultaneously in real time.
https://files.ioc.exchange/iocexchange/media_attachments/files/116/499/861/404/440/171/original/373d7d2f9bf5bc82.png
https://files.ioc.exchange/iocexchange/media_attachments/files/116/499/862/680/763/256/original/483932b71acf3a52.png

2026-05-01 03:44:30 UTC

Weird things that happen when you have a 100GbE pipe to your desk. 8 Gbps of sustained network traffic and the network monitor is like "yeah you're not using much bandwidth"
Also I think there's a 32-bit overflow or something in xfce4-netload-plugin because the rate shows 0.00 Mbps when I get above some threshold (not sure what it is exactly but it's in the 15-40 Gbps range)
https://files.ioc.exchange/iocexchange/media_attachments/files/116/497/247/334/128/502/original/8f0c619c80aec6bf.png

2026-05-01 01:49:36 UTC

- reply

@nprofile…r4z3 @nprofile…r9vg I've described my setup as "qubes on private cloud".
Same idea of isolated vms per security domain except i run them on a shared server and ssh+vnc to them so i can have one mail client, messaging app, etc shared among all of my endpoints

2026-05-01 01:43:34 UTC

- reply

@nprofile…r4z3 @nprofile…r9vg deboan is generally my go-to unless i specifically need something else to test on that distro or something

2026-04-30 16:58:54 UTC

- reply

@nprofile…r9vg I run the vast majority of my stuff that opens a socket to the internet in VMs that I can reboot much more casually.
But also I'm mostly running single-user systems. very few kernel bugs are RCEs, those are major headlines and the kind of thing that *would* get me to do an out of cycle reboot. But something that lets you privesc to kernel while running as a normal user really doesn't matter to me.
If you pop my workstation you a) can get all of my files and data, root doesn't buy you much else and b) it's a dev box I almost always have a root shell open or am going to su/sudo in the next few minutes to do something you can keylog.
The goal is to prevent an attacker from getting code exec as my local user in the first place not preventing root once they do that.

2026-04-30 16:08:40 UTC

- reply

In debian land i'm used to installing packages whenever I need them, patching every month or so, and rebooting ~annually when I have hardware maintenance to do

2026-04-30 16:08:04 UTC

- reply

Do normal arch users just reboot constantly? I'm trying to imagine the UX of installing a random package and having vulkan suddenly stop working until you reboot because your kernel driver and userspace are out of sync or something like that

2026-04-30 16:01:28 UTC

Using Arch as a CI runner has been quite the interesting journey due to their packaging workflow and the fact that installing any new package effectively brings in a full system upgrade with it.
An upgrade that can include some combination of kernel, nvidia kernel module, and userspace Vulkan stack.
Which can result in Vulkan context creation failing with errorIncompatibleDriver until you reboot and everything is in sync again.
So now the "spawn VM" script has special cased logic for Arch that patches, shuts down, and restarts the VM before actually launching the build on it
https://github.com/ngscopeclient/scopehal-ci-scripts/blob/main/vm/spawn-vm

2026-04-29 03:50:19 UTC

- reply

@nprofile…rues Intel XL710 -> Mellanox ConnectX 6. Haven't checked specs but it's likely faster, I think the old NIC was a gen3?

2026-04-29 03:46:30 UTC

Well this was not expected.
The 100G optics I ordered for my desk came in, and swapping the pipe to my desk from 40 to 100G gave a huge improvement in Ceph performance.
But I don't understand *why*.
Baseline config from earlier in the year: client on 40G, cluster nodes on dual 10G, 1558 MB/s on linear reads
Moving cluster nodes to 40G, client on 40G: 1864 MB/s
Moving client to 100G, cluster nodes on 40G: 3787 MB/s.
The confusing thing is, even 3787 MB/s is only about 30 Gbps, so after protocol overhead I would expect it to fit comfortably in 40G. Why can I get this performance with the client on 100G, but not on 40?
https://files.ioc.exchange/iocexchange/media_attachments/files/116/485/423/725/240/394/original/9cc10ec6167dd1c0.png

2026-04-27 19:48:09 UTC

- reply

@nprofile…scs2 silver chloride plus sodium chloride and potassium bitartrate.
I think (don't fully understand the theory of operation) the NaCl improves the extremely poor solubility of silver chloride slightly while the tartrate is ... Doing something to improve deposition and ensure a shiny smooth deposition (apparently after deposition you're supposed to wipe it down with just aqueous potassium bitartrate which somehow cleans the surface)

2026-04-27 19:42:25 UTC

- reply

@nprofile…scs2 I've pretty much standardized on SMA for everything.
My VNA and vector sig gen both have N ports that have permanent SMA converters on them.
The BMAs on my big lecroy scope are converted to 2.92 or SMA any time I don't have an active probe on them.
The BERT has 3 channels of untouched 2.92 on it, but the one channel I use most often for slightly slower work has a 2.92 to SMA adapter on it. At some point I might put a 2.92 connector saver on another channel

2026-04-27 19:23:04 UTC

- reply

@nprofile…scs2 i would prefer sma it's likely getting a n to sma adapter semi permanently installed

2026-04-27 18:51:39 UTC

- reply

@nprofile…scs2 I'm gonna be buying a r&s nrp18t soon but not for work, for my own lab mostly as a cal standard for other equipment

2026-04-27 12:15:33 UTC

Chat is it normal to bring personal standards into work to calibrate their equipment against?

2026-04-27 06:33:02 UTC

Anybody familiar enough with Supermicro main board architecture to know what they're using the little Lattice CPLD for? Just IO expansion for the BMC or what?

2026-04-27 05:48:26 UTC

Testing the silver chloride based plating powder on some PCB samples that I milled.
It plates bare copper as well as, interestingly, ENIG coated surfaces but doesn't seem to deposit on non-metals.
I milled a cavity into layer 2 on this scrap board (ignore the solder paste granules this board has been used for stencil testing too and I didn't bother fully cleaning lol.
I used a combination of a lint-free swab and wooden toothpick to rub the plating composition over the surface. Plating quality deep in the cavity near the edges was defintiely sub-par, you can see the middle area is brighter looking compared to the edges because I was able to rub it more completely. But if I was trying to do rework or wire bonding to this it would probably be good.
Gonna take it into work tomorrow and test if it is in fact wire bondable (and how well it stands up to oxidation etc when heated)
https://files.ioc.exchange/iocexchange/media_attachments/files/116/475/022/391/964/946/original/5e120af59fb33043.jpeg
https://files.ioc.exchange/iocexchange/media_attachments/files/116/475/022/392/910/293/original/ca9363165f0955b2.jpeg

2026-04-27 03:32:13 UTC

- reply

Quick summary from preliminary doc reading having not tried any of them yet:
* VMware does not appear to support virtualizing macos apple silicon guests
* Parallels appears to be subscriptionware but will probably do what I want if I buy the pro edition (seems the lower tier doesn't have the CLI)
* UTM is free and seems to support MacOS guests, so I'm gonna try to install it and see what happens

2026-04-27 02:39:22 UTC

- reply

It looks like the players so far are vmware fusion, UTM, and parallels

2026-04-27 02:38:37 UTC

What's the go to for virtualization on apple silicon macs?
Requirements:
* Free or one time purchase, not subscriptionware
* Can run macos guests with full metal support
Stretch goals:
* Can run arm64 Linux guests with vulkan support

2026-04-26 11:50:12 UTC

- reply

@nprofile…4424 let me put it this way if you are the kind of shop that runs cadence virtuoso on linux, you can afford a RHEL license :p

2026-04-26 11:48:09 UTC

- reply

@nprofile…4424 yes RHEL is what the big eda folks run for linux, fedora is close enough we expect it to work on RHEL despite not officially testing

2026-04-26 11:44:45 UTC

- reply

@nprofile…4424 We do have some, yes. Not sure how many.
Arch has caused me a lot more pain than Fedora to date, mostly due to random package updates causing breakage but also the recent llvmpipe saga causing the GitHub release status to show as failing for the past few months (I might just disable that test suite on GH for now).

2026-04-26 09:35:20 UTC

- reply

So now we have builders for
* arch + nvidia
* debian 13 + nvidia
* debian 12 + nvidia
* ubuntu 26.04 + llvmpipe
* ubuntu 24.04 + nvidia
* win11 + nvidia
Fedora is next on the list as soon as I fix an issue around updating the msys2 runtime on the Windows build...

2026-04-26 09:31:47 UTC

- reply

I ran a few quick calculations on power consumption: measuring at the UPS, the baseline consumption of the lab a minute or so before kicking off a CI job was 3828W.
During testing it hovered around 3900-3950, with a peak of 4007.
Using worst-case numbers assuming (4007-3828) = 179W extra power consumption during the build compared to idle, sustained over 17 minutes, a build would use about 0.05 kWh of power, which as of my last bill would cost me about one cent. Since the actual build is not running at that peak power the entire time, the actual energy requirement is substantially less.

2026-04-26 09:28:56 UTC

OK, the Arch runner is now set up and working after my rather annoying tangle with llvmpipe.
The entire six-job build ran for 17 minutes. I could probably optimize it further: right now the Windows and Ubuntu 24.04 builds are among the slower ones and they share the same GPU forcing them to run sequentially. If I shuffled around which VM ran which job, I could probably fine tune a bit.
Max concurrent jobs was four, three using nvidia cards plus the Ubuntu 26.04 using llvmpipe. I was watching YouTube videos in a SSH+VNC session to another VM on the server the whole time and it didn't stutter or slow down noticeably, suggesting that I wasn't overloading the server.
I think my earlier problems with performance of other VMs during builds were due to the default xcp-ng scheduler granularity being "CPU" not "core"? After changing that things seemed to be a lot better. Was it maybe allocating the entire 32-core CPU to a single VM at a time or something in that mode??

2026-04-26 08:55:14 UTC

- reply

@nprofile…a65n well i dont have easy access to anything else mesa supported on this vm (or any arch instance i have access to) and nvidia doesnt use mesa.

2026-04-26 08:50:56 UTC

- reply

@nprofile…a65n I was able to initialize vulkan and run a couple of shaders without any problems; 13 of my 17 unit tests crashed with weird pointer or memory corruption looking faults in code that seemed to be llvmpipe's JIT binaries from my SPIR-V.
When I put an nvidia card in the same VM, it runs without issue

2026-04-26 08:49:19 UTC

- reply

@nprofile…a65n The crash I was getting was segfaults deep inside LLVM-generated code with no symbols, and a thread named "llvmpipe".
I don't have a full vulkaninfo dump from the VM at that snapshot handy but here's a subset of the info https://dashboard.ngscopeclient.org/tests/920

2026-04-26 08:45:20 UTC

- reply

@nprofile…a65n well i dont even have a display server *installed*, this is a headless VM, so it shouldn't be confused by that

2026-04-26 08:35:20 UTC

- reply

@nprofile…a65n there is no x or wayland server running so i dont think there is any mesa? or is mesa involved in the vulkan stack still even when nothing is being drawn?

2026-04-26 08:34:42 UTC

- reply

@nprofile…a65n I do not have any AMD cards in the VM server at the moment only nvidia.
So my options are currently llvmpipe or a GTX 1630.
Down the road when I shuffle some stuff around and put a 100G NIC in the box, I am considering trying to squeeze an AMD in to enable testing on a larger cross section of platforms

2026-04-26 08:13:17 UTC

Is anybody using llvmpipe on Arch Linux with Vulkan? Have you tested recently?
I'm seeing all kinds of horrible segfaults of trivially simple shaders in ngscopeclient + llvmpipe with Arch on both my own infrastructure and GitHub Actions.
When I put a nvidia card on my Arch VM and use their Vulkan driver instead, everything works fine.

2026-04-23 03:17:56 UTC

- reply

@nprofile…7l00 HIPAA doesn't prohibit collection of information, only how it can be disclosed once it's collected.
A lot of ambulances have cameras in the back too.
There are plenty of legitimate uses besides the "safety/security" justification that's often used; it lets you replay an incident and debrief it to find things you might have missed in the heat of the moment that could have been done better. I know of one service (Parker County) that uses body cam + ambulance cam footage for internal debriefs on all of their cardiac arrest / CPR calls. This helps improve patient outcomes and seems like an entirely reasonable use for the data; it's only being shown to people who were there anyway and is just serving to jog their memory.
If the patient consents to a broader release of the footage, of course, then they can do whatever the patient allows them to.
Which is how you get really cool videos like this https://www.youtube.com/watch?v=p_Fp2hhUPK8 that run through the entire call end to end. They actually have a couple of different ones on this particular call with various analysis and narration by the medics who ran the call, their medical director, and some other folks at the agency.

2026-04-21 22:38:47 UTC

- reply

@nprofile…zwgp as someone who grew up just across the hudson that sounds like a pretty good idea :p

2026-04-21 22:32:58 UTC

The existence of the New York Post implies the existence of the New York GET and HEAD

2026-04-21 15:48:40 UTC

- reply

@nprofile…4424 I have a TBT to 10G SFP+ dongle on my work laptop and keep meaning to put an intel 10G controller asic in a custom framework expansion bay board so i can have SFP+ to the other laptop natively with no dongles

2026-04-21 15:38:14 UTC

- reply

@nprofile…4424 10gbaseT is way worse I don't use it because it's trash

2026-04-21 15:37:59 UTC

- reply

@nprofile…4424 lol oh when i have my switch prototype on a metered lab supply i can very clearly see the few hundred mW from a 1000baseT link toggling

2026-04-21 10:10:24 UTC

- reply

@nprofile…4424 to give you a better idea...
The barely perceptible jump around 21:00 is, I think, me turning on the switch,
Compare this to the much sharper and more well defined jump around 01:00 where I was playing a game on my desktop causing increased CPU/GPU load.
Assuming worst case 75W 24/7 that's 1.8 kWh a day, or about $11 a month at my current power prices. But that's a worst-case number and I expect the actual number will be quite a bit less.
Put a different way, this time of year on a sunny day *one* of my solar panels puts out enough power during daylight hours to run this switch for the entire day, including overnight.
https://files.ioc.exchange/iocexchange/media_attachments/files/116/442/129/582/012/248/original/059d392f6d57ac1c.png

2026-04-21 09:28:19 UTC

- reply

@nprofile…4424 (that should go down now that the sun is shining more)

2026-04-21 09:27:04 UTC

- reply

@nprofile…4424 I mean I have a $800ish power bill it's hard to see a load this small

2026-04-21 09:19:06 UTC

- reply

@nprofile…4424 75w is datasheet max. ups is metered but there's 4 kW of load tens of watts are lost in the noise

2026-04-21 07:37:31 UTC

- reply

I wish "show int status" output on the FS switch showed the vlan number that would be helpful to have at a glance.
https://files.ioc.exchange/iocexchange/media_attachments/files/116/441/539/407/015/835/original/6bca3c967441b9d7.png

2026-04-21 07:30:31 UTC

- reply

I might have to do some tuning on the Ceph cluster to push performance up a bit more, though.
Right now with all 3 cluster nodes and the client on 40G, I get 991 MB/s (7.92 Gbps) of writes, 1864 MB/s (14.9 Gbps) of sequential reads, and 1841 MB/s (14.7 Gbps) of random reads.
This is certainly "usably fast" but not saturating the 40G like I wanted/

2026-04-21 07:27:32 UTC

- reply

There are of course a lot more ports wired up and just not actively lit, I have OM4 all over the house going to wall ports around the lab and office that I can hook things to when experimenting with an FPGA board or something.
But this is a nice near-term upgrade.

2026-04-21 07:25:05 UTC

- reply

After the Ceph node migration I now have only 11 active 10G SFP interfaces on the Nexus, one of which will be going away when I swap the VM server over to 40G.
Of the remaining 10, only 6 are real 10G loads and the other 4 are 1000baseX links to the 1G edge switches.
But once I deploy my own switching for the 1G edge, they'll have real 10/25G uplinks

2026-04-21 07:21:28 UTC

- reply

All wired up, now using 5x 40G + 1x 100G ports for the moment. Office workstation will get bumped from 40 to 100 at some point so then I'll have two free 40G ports, one of which will eventually get hooked to the VM server.
Plus three free 40G ports on the 10G switch.
Also I picked a hostname, it's argon2
https://files.ioc.exchange/iocexchange/media_attachments/files/116/441/480/998/810/783/original/5a71fe4ac4c7006d.jpg

2026-04-21 05:52:40 UTC

- reply

On this switch, trunks default to *no* vlans allowed. You have to do "switchport trunk allowed vlan all" before it will pass any traffic.

2026-04-21 05:31:29 UTC

- reply

Beginning bringup... One of the switches does not like my 40G DAC, it won't link up a trunk. So I'm running a ludicrously short fiber between them instead. Annoying but it works.

2026-04-21 04:11:06 UTC

- reply

Could not have made this power cable more direct lol
https://files.ioc.exchange/iocexchange/media_attachments/files/116/440/736/972/048/647/original/a8ef6eaa60388538.jpg

2026-04-21 03:48:09 UTC

- reply

$DEITY forgive me for I have sinned
https://files.ioc.exchange/iocexchange/media_attachments/files/116/440/646/157/075/161/original/4a36058a0251809b.jpg

2026-04-21 03:24:41 UTC

Getting ready to deploy the new switch and realized a major problem: it doesn't have a hostname yet.
My general theme for switch hostnames is hash algorithms although I've expanded a bit due to exhausting the common ones so I'm using MACs etc as namesakes too.
My current fleet is keccak, blake, skein, poly1305, and ethash. All of the SHA/MD algorithms are previously used.
https://files.ioc.exchange/iocexchange/media_attachments/files/116/440/553/417/582/155/original/783c4571149902dc.jpg

2026-04-20 06:32:40 UTC

- reply

@nprofile…s32f not at all, each of those windows has dozens to hundreds of them. there's hierarchy :p

2026-04-20 06:16:25 UTC

Do I have a problem?
https://files.ioc.exchange/iocexchange/media_attachments/files/116/435/566/289/755/804/original/26891bbf1a1eec55.png

2026-04-20 06:07:58 UTC

- reply

Spider Georg (who lives in a drain not a cave and definitely does not eat 10,000 other spiders a day) came out of the drain for a bit and I got a pic.
Pretty sure he's a hobo spider or giant house spider but not 100%. He can't get out of the tub because the walls are too slippery (I've seen him try) so whatever he is, he's stuck in there lol.
https://files.ioc.exchange/iocexchange/media_attachments/files/116/435/521/250/535/642/original/7fc70a691494e8f0.jpg

2026-04-20 03:11:09 UTC

740.3 kWh month to date. more than we've got during all of March. Solar power season is definitely starting to hit, we're on track to be generating something around 1.2 MWh this month.
Of course this is nowhere near enough to equal my projected consumption of ~4.8 MWh but it'll certainly put a dent in my power bill.

2026-04-20 00:56:13 UTC

- reply

His name is Georg.
Because he is a statistical outlier and should not be counted

2026-04-19 19:46:34 UTC

Our downstairs bathroom has become a de facto half bath since the shower head developed a leak at the elbow that I can't repair without ripping out the water heater and a wall in the laundry room.
This little guy moved into the tub and seems to be doing pretty well for himself. Sometimes he climbs out of the drain and scuttles around the tub but most of the time he seems to like dangling from the drain by two legs.
https://files.ioc.exchange/iocexchange/media_attachments/files/116/433/079/070/286/386/original/f8709759d089d943.jpg

2026-04-19 12:02:42 UTC

- reply

@nprofile…u57d I'm pretty sure the last unsolicited call I got from work was when the low-oxygen alarm in the mechanical room went off at like 5:30am when one lab tech was the only guy in the building.
Luckily it turned out to be the sensor hitting end-of-life and bricking itself rather than going out of calibration (because apparently this sensor had slipped through the cracks and not got the regular maintenance it was supposed to), not an actual N2/Ar leak.

2026-04-19 11:56:54 UTC

- reply

@nprofile…u57d And if it's a major emergency, just go shotgun approach and call/text/email simultaneously and see which gets their attention first.
(That's what my SAR unit uses for their dispatch system, you get mission callouts by SMS, email, and TTS robocall in hopes that one will catch your attention. In a literal life-or-death emergency this is an entirely justifiable level of spam)

2026-04-19 11:54:19 UTC

- reply

@nprofile…u57d and because "be ready to answer the phone and talk to a prospect at any time during normal business hours" is pretty much the job description of a sales rep so it's not like I'm annoying them

2026-04-19 11:53:37 UTC

- reply

@nprofile…u57d If I call a friend or coworker outside of a prearranged meeting/discussion, it's because I have a matter that requires *immediate attention* for the most part.
Businesses, or people acting on behalf of a business like sales reps, are a different story. I do sometimes call rather than emailing because I'm usually looking to have a discussion that will lead to them sending me a quote or something, and the call will get it out of the way quicker than a bunch of back and forth asynchronous messaging (and most businesses aren't reachable by SMS or signal)

2026-04-19 11:46:14 UTC

- reply

Ok all reassembled and patched and took care of some other housekeeping plus reopened most of the wip stuff i wanted to keep active.
Now it's almost 5am and I'm about where I wanted to be at 2200 last night (pcie all shuffled, 100g nic installed in last cluster node, desktops functional)

2026-04-19 10:12:24 UTC

- reply

And the second one is now POSTing after the BIOS reflash too.
Just gotta put the GPU and rest of the RAM back in that I took out while troubleshooting.
Seriously though, *how*???

2026-04-19 09:59:27 UTC

- reply

The machine hanging during POST is on BIOS 1.1 which is even older (from 2021). I had never had a reason to reflash it and while I do usually run fwupd from time to time, maybe I forgot on this machine or supermicro didnt push updates?
Here's hoping that's the problem.

2026-04-19 09:49:29 UTC

- reply

No root cause at this point; working hypothesis is that maybe before it hit the null, whatever went wrong in the Ceph driver might have also scribbled over EFI variables or SMM memory or something that put the firmware into a bad state but that the reflash cleared to a clean slate.

2026-04-19 09:48:45 UTC

Progress on my latest round of craziness in the lab having problems that are Very Computer.
To recap, the sequence of events was:
1) Swap 100G NICs to different PCIe slots in storage cluster nodes 3 and 2 without incident
2) Add 100G NIC to storage cluster node 1 (the current active metadata server)
3) Office, soldering bench, and microscope bench workstations hang with a kernel stack trace from a null deref in the Ceph client driver as soon as node 1 drops offline.
4) After hard power cycling, office workstation (2x Xeon 6144 in Supermicro X11DAi-N) comes up without issue.
Soldering bench (Xeon 8362 in Supermicro X12SPi-TF) and microscope bench (Xeon 5320 in Supermicro X12SPi-TF) fail to boot. Soldering bench machine hangs during POST initializing the USB controller while microscope bench machine POSTs and can get into setup or boot menu, but is unable to launch a bootloader either via removable media or internal SSD.
5) Lots of head scratching and confusion as to how a network filesystem driver segfault could make a machine not POST
After trying everything from removing DIMMs to re-torquing the CPU I pretty much gave up on it being a hardware problem and, at a friend's suggestion, reflashed the BIOS on the microscope bench machine via the BMC (updating from v2.3 to 2.5)
It recovered and now boots, I'm posting from it. Fingers crossed the other machine can be saved the same way.

2026-04-19 04:59:36 UTC

Poking around a bit more at the PCIe configuration on my storage cluster nodes and between the UEFI config and the manual I think I fully understand the topology now.
(do consumer mainboards ever publish this kind of info? I'm gonna guess no... but I havent bought one in ~10 years)
The CPU itself has 48 PCIe lanes, which was the most available when I deployed these (icelake-SP bumped this up to 64, and sapphire rapids to 80).
They're divided into three x16 root complexes.
Group 1 goes into a mux (Br3) which either connects to slot 1 as x16, or bifurcates to x4 to each of the four M.2 ports.
Group 2 goes into a Broadcom PCIe switch which splits off to four slots 5:2. These slots are paired, and each pair (2/3 and 4/5) can be either dual x8s or a single x16, but all four share the single x16 to the root.
Group 3 goes to a mux (Br2) which either connects to slot 7 as x16, or slots 6/7 as dual x8. It can also bifurcate to quad x4+x4+x4+x4 in slot 7 or dual x4+x4 in each of 6/7.
https://files.ioc.exchange/iocexchange/media_attachments/files/116/429/518/204/191/700/original/1a697eb27094959d.png