Andrew Zonenberg

Security and open source at the hardware/software interface. Embedded sec @ IOActive. Lead dev of ngscopeclient/libscopehal. GHz probe designer. Open source networking hardware. "So others may live"

Toots searchable on tootfinder.

Public Key

npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd

NIP-05 Address

azonenberg@ioc-exchange.mostr.pub

Profile Code

nprofile1qqsvxk504cz6sw9cvdc8sqamrl0uljanntw0q8hp48799m6ynxxwfsqpz4mhxue69uhhyetvv9ujuerfw36x7tnsw43qfqrw8c

Publishing to

relay.ditto.pub

Author Public Key

npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd

Show more details

Last Notes

2026-07-04 21:06:57 UTC

- reply

@nprofile…vnlz I really do wonder how the T&M industry stagnated so much to the point that everything I'm doing is super groundbreaking.
Like this seems like an obvious way to do things

2026-07-04 21:05:32 UTC

- reply

@nprofile…vnlz well mostly it would also enable deskewing *arbitrary* LAs not just ILAs.
Or ILAs to analog scopes, etc. Basically just extending the existing cross-correlation based deskew logic to support non-analog signals

2026-07-04 21:04:02 UTC

- reply

@nprofile…vnlz see https://ioc.exchange/@azonenberg/116856901135293906 for an example of a cross-trigger between an ILA (primary) and external analog scope (secondary).
There's no way to do this with any other debug flow I've ever seen - like you can certainly use a xilinx ILA and a GPIO to trigger another scope but you can't view the data all on one timeline like this

2026-06-30 19:42:22 UTC

Chat what kind of fiber connector is this
https://files.ioc.exchange/iocexchange/media_attachments/files/116/840/757/427/374/824/original/3d67f9e9c120836c.jpg
https://files.ioc.exchange/iocexchange/media_attachments/files/116/840/757/860/604/210/original/5010bff56b06a7b6.jpg
https://files.ioc.exchange/iocexchange/media_attachments/files/116/840/758/209/634/727/original/b37dabcf355e6307.jpg

2026-06-19 05:34:25 UTC

- reply

@nprofile…5yze Sadly neither exists in the US to my knowledge, although the parent company may well sell the same product here under a different brand.
The one that caused the issue was an amazonbasics product at school labeled "latex free" although with amazonbasics who knows how accurate that is lol.
Couldn't find any obvious labels on the box as to if the adhesive was acrylic based or silicone based or whatever.

2026-06-19 00:04:31 UTC

Do all bandaids use the same type of adhesive or are there multiple kinds in use? Kid seems to be allergic to at least some of them.

2026-06-01 15:54:51 UTC

- reply

@nprofile…lqye that's what I'm doing, but in this case I was interpolating sub sample level crossings for measuring rise time of a signal and using float temporaries since the interpolation code worked in floating point domain

2026-06-01 15:36:12 UTC

- reply

(float64 is woefully inadequate for trying to measure sub picosecond time resolution over even a few hundred μs)

2026-05-29 23:42:56 UTC

- reply

@nprofile…x8t7 sure you might find some drug kingpin or something who just wants to have $1M in gold in his basement to show off to his underlings, but I doubt they would want a huge amount of gold either, because then they'd have the same problem of turning it into something useful

2026-05-29 23:41:53 UTC

- reply

@nprofile…x8t7 what i'm more curious about is, who *buys* millions of dollars of gold?
Like if some guy in a trench coat just handed me $20M in gold I have no idea how I'd turn that into cash without raising a lot of eyebrows.
Short of a central bank or industrial scale jewelry manufacturer or something, who is going to buy that? Your average neighborhood "we buy gold" pawnshop certainly isn't going to have $20M lying around. Folks who have legitimate uses for large quantities of gold already have established supply chains and will definitely be asking questions if a stranger walks up offering a huge amount.

2026-05-28 09:20:50 UTC

- reply

@nprofile…evsw i found out when i logged into the second and openssh already had the fingerprint in known hosts

2026-05-28 09:16:29 UTC

- reply

@nprofile…evsw same keys out of the box with no config.
i've figured out how to roll the rsa key in config, not the ecdsa/ed25519 yet

2026-05-28 08:42:28 UTC

- reply

These are the keys of the other switch that I haven't rekeyed yet (so whatever shipped in the firmware):
| ssh-hostkey:
| 2048 5b:4b:c3:16:59:b5:96:91:f3:a2:3c:4c:bc:cc:65:a7 (RSA)
| 256 ef:d7:50:22:01:ac:4c:25:24:ff:f5:40:b9:1d:db:5d (ECDSA)
|_ 256 f3:18:f6:7b:8f:6f:19:83:35:90:26:60:46:66:8c:43 (ED25519)

2026-05-28 08:40:59 UTC

FS switch SSH update: the SSH server appears to advertise RSA, ed25519, and ECDSA host keys.
I can change the RSA key with "rsa key NAME generate 4096" and "ip ssh server host-key rsa key NAME".
I have so far not figured out how to change the ed25519 or ECDSA keys.

2026-05-25 01:51:57 UTC

- reply

@nprofile…zaqv 70 pounds for just one side? No thanks lol

2026-05-24 20:14:47 UTC

- reply

1u rack panel full of bnc connectors (i have 2 but keeping one)
https://files.ioc.exchange/iocexchange/media_attachments/files/116/631/383/107/684/144/original/a45853820c800c32.jpg

2026-05-24 20:13:19 UTC

- reply

Assorted boards with spartan-3a fpgas on them, I've probably lost the schematics but they're 4 layer so should be easy to reverse or just salvage the chips
https://files.ioc.exchange/iocexchange/media_attachments/files/116/631/377/037/714/861/original/fb0c293b68a4ef2f.jpg

2026-05-24 20:10:57 UTC

- reply

Little tool organizer box
https://files.ioc.exchange/iocexchange/media_attachments/files/116/631/367/416/332/286/original/5cbf1b0608de3393.jpg

2026-05-24 20:09:38 UTC

- reply

More sata cables than i know what to do with, I'm keeping a few but please take some off my hands
https://files.ioc.exchange/iocexchange/media_attachments/files/116/631/363/326/000/077/original/abbca8dfc1e32e14.jpg

2026-05-16 19:17:44 UTC

- reply

@nprofile…fqfm @nprofile…cy5p I'm hoping one day a bunch of angry nerds will design either an entire suite of new electronics/controls for a popular late model vehicle, or better yet an entirely new vehicle, that is simple, low tech, repairable, and open source to the extent practical.
Rip out the CAN bus controlled door locks and replace them with ones that use physical keys that don't need batteries.
Remove the touchscreen HVAC controls and put in physical knobs.
No fly-by-wire anywhere except where required to perform a key safety function like ABS. If you have to put software between human control inputs and steering/braking/acceleration, it should be as minimal as possible, formally verified, and multiply redundant processors, sensors, and actuators.
Publish full mechanical drawings of common wear components to make third party replacements easier and allow the vehicles to outlive the manufacturers if they later go out of business.
No ADAS features whatsoever.

2026-05-16 18:26:21 UTC

- reply

@nprofile…cy5p I am absolutely prepared to excise radio power amplifiers with a hot air rework station if it comes down to that level though.

2026-05-16 18:25:45 UTC

- reply

@nprofile…cy5p If I am ever forced to buy a 2015+ vehicle (or whenever they started putting telematics on cars) due to lack of alternatives, you can bet I am going to go bumper to bumper with a spectrum analyzer and nearfield probe set and neutralize every radio transmitter on the car by any means necessary.
Removing the entire ECU? Cutting traces to the antenna and replacing them with a 50-ohm terminator? High voltage into the antenna until it stops responding?
One way or another, it won't phone home. But the longer it takes and the more work it requires, the more pissed I'll be.

2026-05-16 18:23:46 UTC

- reply

@nprofile…cy5p I work in computer security and do a lot of work on automotive stuff.
People ask me what I think of self-driving cars.
I tell them I get to work on a bicycle. The smartest thing on it is the USB charge controller for my headlight and if it makes a funny noise or starts smoking I can yank it off the handlebars and yeet it somewhere far away from me.

2026-05-16 01:51:31 UTC

Load break cutouts on a cross connect downtown
https://files.ioc.exchange/iocexchange/media_attachments/files/116/581/723/195/724/440/original/9289c8195bbe08b8.jpg

2026-05-15 22:42:57 UTC

- reply

@nprofile…u6vt yeah a good trainer will recognize when someone is at the breaking point and pull them before they get hurt, or set a max number of reps. you're not trying to kill the team you want to build them up

2026-05-15 22:34:07 UTC

- reply

As written this probably wouldn't work but it's inspired by the military / sport training practice of doing a group run across a field or something.
The first one to the other side gets to take a shower. Everyone else does it again.
The idea isn't to punish the worse performers, it's to give more practice/exercise to those who are underperforming until everyone is able to meet the standard eventually.

2026-05-15 22:31:18 UTC

Cursed exam idea: everyone gets the same problem, first to submit a correct answer gets a passing grade.
Everyone else gets another problem.
The exam will continue until everyone passes.

2026-05-15 03:04:08 UTC

Fun engineering/privacy problem: design a device that plugs into an ESP32 based sensor node that has a microphone and reports ambient noise levels in dBa via MQTT or similar.
That's the easy part.
The hard part is to do so in a way that you can prove it's incapable of being used as a bug/listening device even in the worst case scenario (attacker running arbitrary code on any programmable component).
My initial thought is to record audio from a MEMS mic into a tiny MCU with limited RAM capacity, then report results to the main system processor through some kind of R-C low pass filter that only lets like one bit per second pass through.
Or maybe a bank of analog filters and peak hold blocks? I want proper weighted dBa.

2026-05-13 06:00:40 UTC

Progress! CI jobs for two consecutive commits dispatched by GitHub commit hook.
First one had some minor issues, I fixed the CI script, then pushed a new commit before the first build had finished running.
If everything works, we should get a full build of both commits soon.
I've started work on the infrastructure for artifact uploading but need another day or two to get that all wrapped up.
https://files.ioc.exchange/iocexchange/media_attachments/files/116/565/734/298/291/923/original/2091e189e76c2118.png

2026-05-13 00:26:12 UTC

- reply

@nprofile…thd8 that sounds like something a scopehal filter could do. You'd need to write a boundary scan decoder to go from the jtag scans to digital waveforms for each IO, but then you could run the existing spi decodes on top of that.
Complex chains of tunneled protocols is something we can handle with ease given the right intermediate blocks

2026-05-12 22:46:21 UTC

It's been a while since I've done web stuff. Did I screw up anything too horribly here?
https://github.com/ngscopeclient/scopehal-ci-scripts/blob/main/api/github-hook.php
This is a web hook that is only triggered by the 'push' event on ngscopeclient/scopehal-apps.
Goal is to ensure that
a) nobody but github can trigger builds (to prevent DoSing the CI platform with a ridiculous number of builds etc)
b) a compromise of GitHub's webhook infrastructure can trigger builds of real commits in the repo, but not run arbitrary shell commands on the CI runner or pull from an untrusted fork (i.e. no shell command injections etc in the branch/commit strings which will be fed to a zillion bash scripts downstream)

2026-05-12 19:32:26 UTC

- reply

@nprofile…ffys seriously, i love raw vulkan with minimal wrappers for compute acceleration... it's almost as low level as CUDA but it's not absurdly heavyweight tooling wise and is portable to anything from a Haswell-era integrated GPU to a modern nvidia monster to a macbook air

2026-05-12 19:31:13 UTC

- reply

@nprofile…ffys apt install libvulkan-dev
Problem solved?

2026-05-09 13:41:59 UTC

- reply

@nprofile…t9xu if i did somehow have ~4x the generation capacity i in fact do, I'd have a different problem though... Anything that can store tens of MWh of extra energy can let it out again if something goes wrong.
But that's a problem i would save for somebody else because I'm on a net metering plan that resets at the end of March which conveniently aligns with when solar production really picks up for the year.
So if i had a ~72 kW peak solar farm instead of the 18 I do in reality, I'd just throw my extra power at the utility to use for other customers and draw down my negative bill balance over the winter. The vast majority of our power here in WA is renewable anyway (hydro and wind for the most part)

2026-05-09 08:13:11 UTC

- reply

@nprofile…t9xu I burn all of my surplus overnight, there's nothing to save for later anyway

2026-05-08 18:24:49 UTC

398 kWh solar generation for May so far and it's not even noon on the 8th.
This is more power than I generated during all of February (393 kWh) lol. Solar season is definitely a thing.

2026-05-07 12:00:18 UTC

Hmmm... How likely are platform specific memory safety issues in ngscopeclient/libscopehal? I'm thinking "not very" since almost all of the OS-specific stuff code is in third party portability libraries.
I ask because right now we do all of our CI builds as optimized release builds with the intention of creating nightly builds other people can download and play with.
But I want to do ASAN+debug builds to catch memory safety issues during tests (there was at least one that this could have caught in the past).
The question is, do we double up *all* of the CI builds, using twice as much time/power? Or is it sufficient to do a *single* debug/sanitizer build, on one supported platform (probably one using llvmpipe to avoid monopolizing a real GPU)?

2026-05-06 21:01:54 UTC

- reply

I was hoping I'd be sub $700 for April but didn't quite generate enough juice to do it.
May started out strong with four days of >60 kWh solar generation (which counted as part of the April billing cycle) before some clouds rolled in, so we'll see... May around here tends to be the transition between the monsoon season and cloudless summer weather.
Hoping to hit below 3 MWh imported in May, so let's see how that goes.
Here's some graphs from my analysis spreadsheet showing my demand vs utility rates over the last few years.
https://files.ioc.exchange/iocexchange/media_attachments/files/116/529/636/620/185/035/original/c9af5918369a39b5.png
https://files.ioc.exchange/iocexchange/media_attachments/files/116/529/639/468/136/893/original/324b3adc555abe65.png

2026-05-06 20:56:35 UTC

Got the official power numbers from my utility for the April billing cycle.
3572 kWh imported from grid, 200 kWh overproduction exported, for net 3372 kWh consumed/billed.
That is an all-time record for lowest power consumption (at least back to January 2024 which is the oldest bill I have easy access to).
Too bad it's also an all-time *high* for price per kWh - the effective price per kWh after taxes and fees hit 21.7 cents, a 15.4% increase since the start of the year and 53.9% since January 2024.
Even so, the $731 bill for 3372 kWh is the smallest since I paid $722 for 3983 kWh back in July 2025. And that was an outlier, going further back the next time I had a bill that small was $618 in October 2024 for 3998 kWh.

2026-05-04 18:17:53 UTC

- reply

@nprofile…02n5 something something taylor swift private jet

2026-05-04 09:33:59 UTC

- reply

For comparison... my 16 GHz LeCroy oscilloscope puts out 40 Gsps * 4 channels * 8 bits of raw ADC samples, not counting the flatness corrections done in gateware/firmware.
That's 160 GB/s or 1.28 Tbps of raw samples.
That would even fit in NVLink 2.0 much less the current gen4/5 stuff.
Imagine four channels of 16 GHz bandwidth waveform data straight into a (very large) GPU nonstop... We'd have to do a hell of a lot of optimization to ngscopeclient to keep up and probably add multi-GPU support but it would be so much fun lol.

2026-05-04 09:29:03 UTC

You know what would be really nice (but nobody is ever going to build)?
Oscilloscope that replaces the ungodly slow USB3/1000baseT PC interface port with NVLink.
Forget PCIe and Thunderbolt... 900 GB/s of bandwidth straight from the ADC to my GPU? Sign me up.

2026-05-03 18:51:26 UTC

Whenever me and my wife are driving somewhere together and are backing out of a driveway, making a turn at a sketchy intersection, etc whoever is in the passenger seat always acts as a second set of eyes calling out traffic or "clear right" etc as appropriate.
From what I've seen this is common practice in military and fire department circles but pretty rare in the civilian world for some reason. Not sure why, better situational awareness definitely makes things safer.

2026-05-02 15:17:16 UTC

- reply

@nprofile…a65n it claimed submersible and i certainly got it pretty wet over the years. Pretty sure it's been submerged at least once.

2026-05-02 14:25:21 UTC

- reply

@nprofile…a65n air quality + IP rated might be harder since you'll need the sensor exposed

2026-05-02 14:03:39 UTC

- reply

@nprofile…a65n doesn't exist afaik

2026-05-02 08:57:47 UTC

I have a pair of end-of-life phones (CAT S61 and S62) with FLIR Lepton thermal cores in them.
Anybody know if there is a good way to disassemble them to salvage the thermal modules without destroying them? I don't care about the rest of the phone, it's gonna end up in ewaste.

2026-05-02 05:20:15 UTC

I just tried to ctrl-O to save a file in a GUI text editor.
I think I've been spending too much time in nano lately

2026-05-01 16:41:37 UTC

- reply

@nprofile…3tsk lol moss is native to this area the grass is probably what doesn't belong

2026-05-01 16:36:14 UTC

- reply

@nprofile…3tsk the yard is like half moss already I just need to get rid of the grass and buttercups lol

2026-05-01 16:35:47 UTC

- reply

@nprofile…3tsk i would like to replace our grass with moss... Never needs mowing, nice and soft

2026-05-01 14:48:34 UTC

Having a bit of fun with my 100G LAN, new GPU, and a pair of ThunderScopes.
Here's not one but *two* 1 Gsps live waveform streams being FFT'd simultaneously in real time.
https://files.ioc.exchange/iocexchange/media_attachments/files/116/499/861/404/440/171/original/373d7d2f9bf5bc82.png
https://files.ioc.exchange/iocexchange/media_attachments/files/116/499/862/680/763/256/original/483932b71acf3a52.png

2026-05-01 03:44:30 UTC

Weird things that happen when you have a 100GbE pipe to your desk. 8 Gbps of sustained network traffic and the network monitor is like "yeah you're not using much bandwidth"
Also I think there's a 32-bit overflow or something in xfce4-netload-plugin because the rate shows 0.00 Mbps when I get above some threshold (not sure what it is exactly but it's in the 15-40 Gbps range)
https://files.ioc.exchange/iocexchange/media_attachments/files/116/497/247/334/128/502/original/8f0c619c80aec6bf.png

2026-05-01 01:49:36 UTC

- reply

@nprofile…r4z3 @nprofile…r9vg I've described my setup as "qubes on private cloud".
Same idea of isolated vms per security domain except i run them on a shared server and ssh+vnc to them so i can have one mail client, messaging app, etc shared among all of my endpoints

2026-05-01 01:43:34 UTC

- reply

@nprofile…r4z3 @nprofile…r9vg deboan is generally my go-to unless i specifically need something else to test on that distro or something

2026-04-30 16:58:54 UTC

- reply

@nprofile…r9vg I run the vast majority of my stuff that opens a socket to the internet in VMs that I can reboot much more casually.
But also I'm mostly running single-user systems. very few kernel bugs are RCEs, those are major headlines and the kind of thing that *would* get me to do an out of cycle reboot. But something that lets you privesc to kernel while running as a normal user really doesn't matter to me.
If you pop my workstation you a) can get all of my files and data, root doesn't buy you much else and b) it's a dev box I almost always have a root shell open or am going to su/sudo in the next few minutes to do something you can keylog.
The goal is to prevent an attacker from getting code exec as my local user in the first place not preventing root once they do that.

2026-04-30 16:08:40 UTC

- reply

In debian land i'm used to installing packages whenever I need them, patching every month or so, and rebooting ~annually when I have hardware maintenance to do

2026-04-30 16:08:04 UTC

- reply

Do normal arch users just reboot constantly? I'm trying to imagine the UX of installing a random package and having vulkan suddenly stop working until you reboot because your kernel driver and userspace are out of sync or something like that

2026-04-30 16:01:28 UTC

Using Arch as a CI runner has been quite the interesting journey due to their packaging workflow and the fact that installing any new package effectively brings in a full system upgrade with it.
An upgrade that can include some combination of kernel, nvidia kernel module, and userspace Vulkan stack.
Which can result in Vulkan context creation failing with errorIncompatibleDriver until you reboot and everything is in sync again.
So now the "spawn VM" script has special cased logic for Arch that patches, shuts down, and restarts the VM before actually launching the build on it
https://github.com/ngscopeclient/scopehal-ci-scripts/blob/main/vm/spawn-vm

2026-04-29 03:50:19 UTC

- reply

@nprofile…rues Intel XL710 -> Mellanox ConnectX 6. Haven't checked specs but it's likely faster, I think the old NIC was a gen3?

2026-04-29 03:46:30 UTC

Well this was not expected.
The 100G optics I ordered for my desk came in, and swapping the pipe to my desk from 40 to 100G gave a huge improvement in Ceph performance.
But I don't understand *why*.
Baseline config from earlier in the year: client on 40G, cluster nodes on dual 10G, 1558 MB/s on linear reads
Moving cluster nodes to 40G, client on 40G: 1864 MB/s
Moving client to 100G, cluster nodes on 40G: 3787 MB/s.
The confusing thing is, even 3787 MB/s is only about 30 Gbps, so after protocol overhead I would expect it to fit comfortably in 40G. Why can I get this performance with the client on 100G, but not on 40?
https://files.ioc.exchange/iocexchange/media_attachments/files/116/485/423/725/240/394/original/9cc10ec6167dd1c0.png

2026-04-27 19:48:09 UTC

- reply

@nprofile…scs2 silver chloride plus sodium chloride and potassium bitartrate.
I think (don't fully understand the theory of operation) the NaCl improves the extremely poor solubility of silver chloride slightly while the tartrate is ... Doing something to improve deposition and ensure a shiny smooth deposition (apparently after deposition you're supposed to wipe it down with just aqueous potassium bitartrate which somehow cleans the surface)

2026-04-27 19:42:25 UTC

- reply

@nprofile…scs2 I've pretty much standardized on SMA for everything.
My VNA and vector sig gen both have N ports that have permanent SMA converters on them.
The BMAs on my big lecroy scope are converted to 2.92 or SMA any time I don't have an active probe on them.
The BERT has 3 channels of untouched 2.92 on it, but the one channel I use most often for slightly slower work has a 2.92 to SMA adapter on it. At some point I might put a 2.92 connector saver on another channel

2026-04-27 19:23:04 UTC

- reply

@nprofile…scs2 i would prefer sma it's likely getting a n to sma adapter semi permanently installed

2026-04-27 18:51:39 UTC

- reply

@nprofile…scs2 I'm gonna be buying a r&s nrp18t soon but not for work, for my own lab mostly as a cal standard for other equipment

2026-04-27 12:15:33 UTC

Chat is it normal to bring personal standards into work to calibrate their equipment against?

2026-04-27 06:33:02 UTC

Anybody familiar enough with Supermicro main board architecture to know what they're using the little Lattice CPLD for? Just IO expansion for the BMC or what?

2026-04-27 05:48:26 UTC

Testing the silver chloride based plating powder on some PCB samples that I milled.
It plates bare copper as well as, interestingly, ENIG coated surfaces but doesn't seem to deposit on non-metals.
I milled a cavity into layer 2 on this scrap board (ignore the solder paste granules this board has been used for stencil testing too and I didn't bother fully cleaning lol.
I used a combination of a lint-free swab and wooden toothpick to rub the plating composition over the surface. Plating quality deep in the cavity near the edges was defintiely sub-par, you can see the middle area is brighter looking compared to the edges because I was able to rub it more completely. But if I was trying to do rework or wire bonding to this it would probably be good.
Gonna take it into work tomorrow and test if it is in fact wire bondable (and how well it stands up to oxidation etc when heated)
https://files.ioc.exchange/iocexchange/media_attachments/files/116/475/022/391/964/946/original/5e120af59fb33043.jpeg
https://files.ioc.exchange/iocexchange/media_attachments/files/116/475/022/392/910/293/original/ca9363165f0955b2.jpeg

2026-04-27 03:32:13 UTC

- reply

Quick summary from preliminary doc reading having not tried any of them yet:
* VMware does not appear to support virtualizing macos apple silicon guests
* Parallels appears to be subscriptionware but will probably do what I want if I buy the pro edition (seems the lower tier doesn't have the CLI)
* UTM is free and seems to support MacOS guests, so I'm gonna try to install it and see what happens

2026-04-27 02:39:22 UTC

- reply

It looks like the players so far are vmware fusion, UTM, and parallels

2026-04-27 02:38:37 UTC

What's the go to for virtualization on apple silicon macs?
Requirements:
* Free or one time purchase, not subscriptionware
* Can run macos guests with full metal support
Stretch goals:
* Can run arm64 Linux guests with vulkan support

2026-04-26 11:50:12 UTC

- reply

@nprofile…4424 let me put it this way if you are the kind of shop that runs cadence virtuoso on linux, you can afford a RHEL license :p

2026-04-26 11:48:09 UTC

- reply

@nprofile…4424 yes RHEL is what the big eda folks run for linux, fedora is close enough we expect it to work on RHEL despite not officially testing

2026-04-26 11:44:45 UTC

- reply

@nprofile…4424 We do have some, yes. Not sure how many.
Arch has caused me a lot more pain than Fedora to date, mostly due to random package updates causing breakage but also the recent llvmpipe saga causing the GitHub release status to show as failing for the past few months (I might just disable that test suite on GH for now).

2026-04-26 09:35:20 UTC

- reply

So now we have builders for
* arch + nvidia
* debian 13 + nvidia
* debian 12 + nvidia
* ubuntu 26.04 + llvmpipe
* ubuntu 24.04 + nvidia
* win11 + nvidia
Fedora is next on the list as soon as I fix an issue around updating the msys2 runtime on the Windows build...

2026-04-26 09:31:47 UTC

- reply

I ran a few quick calculations on power consumption: measuring at the UPS, the baseline consumption of the lab a minute or so before kicking off a CI job was 3828W.
During testing it hovered around 3900-3950, with a peak of 4007.
Using worst-case numbers assuming (4007-3828) = 179W extra power consumption during the build compared to idle, sustained over 17 minutes, a build would use about 0.05 kWh of power, which as of my last bill would cost me about one cent. Since the actual build is not running at that peak power the entire time, the actual energy requirement is substantially less.

2026-04-26 09:28:56 UTC

OK, the Arch runner is now set up and working after my rather annoying tangle with llvmpipe.
The entire six-job build ran for 17 minutes. I could probably optimize it further: right now the Windows and Ubuntu 24.04 builds are among the slower ones and they share the same GPU forcing them to run sequentially. If I shuffled around which VM ran which job, I could probably fine tune a bit.
Max concurrent jobs was four, three using nvidia cards plus the Ubuntu 26.04 using llvmpipe. I was watching YouTube videos in a SSH+VNC session to another VM on the server the whole time and it didn't stutter or slow down noticeably, suggesting that I wasn't overloading the server.
I think my earlier problems with performance of other VMs during builds were due to the default xcp-ng scheduler granularity being "CPU" not "core"? After changing that things seemed to be a lot better. Was it maybe allocating the entire 32-core CPU to a single VM at a time or something in that mode??

2026-04-26 08:55:14 UTC

- reply

@nprofile…a65n well i dont have easy access to anything else mesa supported on this vm (or any arch instance i have access to) and nvidia doesnt use mesa.

2026-04-26 08:50:56 UTC

- reply

@nprofile…a65n I was able to initialize vulkan and run a couple of shaders without any problems; 13 of my 17 unit tests crashed with weird pointer or memory corruption looking faults in code that seemed to be llvmpipe's JIT binaries from my SPIR-V.
When I put an nvidia card in the same VM, it runs without issue

2026-04-26 08:49:19 UTC

- reply

@nprofile…a65n The crash I was getting was segfaults deep inside LLVM-generated code with no symbols, and a thread named "llvmpipe".
I don't have a full vulkaninfo dump from the VM at that snapshot handy but here's a subset of the info https://dashboard.ngscopeclient.org/tests/920

2026-04-26 08:45:20 UTC

- reply

@nprofile…a65n well i dont even have a display server *installed*, this is a headless VM, so it shouldn't be confused by that

2026-04-26 08:35:20 UTC

- reply

@nprofile…a65n there is no x or wayland server running so i dont think there is any mesa? or is mesa involved in the vulkan stack still even when nothing is being drawn?

2026-04-26 08:34:42 UTC

- reply

@nprofile…a65n I do not have any AMD cards in the VM server at the moment only nvidia.
So my options are currently llvmpipe or a GTX 1630.
Down the road when I shuffle some stuff around and put a 100G NIC in the box, I am considering trying to squeeze an AMD in to enable testing on a larger cross section of platforms

2026-04-26 08:13:17 UTC

Is anybody using llvmpipe on Arch Linux with Vulkan? Have you tested recently?
I'm seeing all kinds of horrible segfaults of trivially simple shaders in ngscopeclient + llvmpipe with Arch on both my own infrastructure and GitHub Actions.
When I put a nvidia card on my Arch VM and use their Vulkan driver instead, everything works fine.

2026-04-23 03:17:56 UTC

- reply

@nprofile…7l00 HIPAA doesn't prohibit collection of information, only how it can be disclosed once it's collected.
A lot of ambulances have cameras in the back too.
There are plenty of legitimate uses besides the "safety/security" justification that's often used; it lets you replay an incident and debrief it to find things you might have missed in the heat of the moment that could have been done better. I know of one service (Parker County) that uses body cam + ambulance cam footage for internal debriefs on all of their cardiac arrest / CPR calls. This helps improve patient outcomes and seems like an entirely reasonable use for the data; it's only being shown to people who were there anyway and is just serving to jog their memory.
If the patient consents to a broader release of the footage, of course, then they can do whatever the patient allows them to.
Which is how you get really cool videos like this https://www.youtube.com/watch?v=p_Fp2hhUPK8 that run through the entire call end to end. They actually have a couple of different ones on this particular call with various analysis and narration by the medics who ran the call, their medical director, and some other folks at the agency.

2026-04-21 22:38:47 UTC

- reply

@nprofile…zwgp as someone who grew up just across the hudson that sounds like a pretty good idea :p

2026-04-21 22:32:58 UTC

The existence of the New York Post implies the existence of the New York GET and HEAD

2026-04-21 15:48:40 UTC

- reply

@nprofile…4424 I have a TBT to 10G SFP+ dongle on my work laptop and keep meaning to put an intel 10G controller asic in a custom framework expansion bay board so i can have SFP+ to the other laptop natively with no dongles

2026-04-21 15:38:14 UTC

- reply

@nprofile…4424 10gbaseT is way worse I don't use it because it's trash

2026-04-21 15:37:59 UTC

- reply

@nprofile…4424 lol oh when i have my switch prototype on a metered lab supply i can very clearly see the few hundred mW from a 1000baseT link toggling

2026-04-21 10:10:24 UTC

- reply

@nprofile…4424 to give you a better idea...
The barely perceptible jump around 21:00 is, I think, me turning on the switch,
Compare this to the much sharper and more well defined jump around 01:00 where I was playing a game on my desktop causing increased CPU/GPU load.
Assuming worst case 75W 24/7 that's 1.8 kWh a day, or about $11 a month at my current power prices. But that's a worst-case number and I expect the actual number will be quite a bit less.
Put a different way, this time of year on a sunny day *one* of my solar panels puts out enough power during daylight hours to run this switch for the entire day, including overnight.
https://files.ioc.exchange/iocexchange/media_attachments/files/116/442/129/582/012/248/original/059d392f6d57ac1c.png

2026-04-21 09:28:19 UTC

- reply

@nprofile…4424 (that should go down now that the sun is shining more)

2026-04-21 09:27:04 UTC

- reply

@nprofile…4424 I mean I have a $800ish power bill it's hard to see a load this small

2026-04-21 09:19:06 UTC

- reply

@nprofile…4424 75w is datasheet max. ups is metered but there's 4 kW of load tens of watts are lost in the noise

2026-04-21 07:37:31 UTC

- reply

I wish "show int status" output on the FS switch showed the vlan number that would be helpful to have at a glance.
https://files.ioc.exchange/iocexchange/media_attachments/files/116/441/539/407/015/835/original/6bca3c967441b9d7.png

2026-04-21 07:30:31 UTC

- reply

I might have to do some tuning on the Ceph cluster to push performance up a bit more, though.
Right now with all 3 cluster nodes and the client on 40G, I get 991 MB/s (7.92 Gbps) of writes, 1864 MB/s (14.9 Gbps) of sequential reads, and 1841 MB/s (14.7 Gbps) of random reads.
This is certainly "usably fast" but not saturating the 40G like I wanted/

2026-04-21 07:27:32 UTC

- reply

There are of course a lot more ports wired up and just not actively lit, I have OM4 all over the house going to wall ports around the lab and office that I can hook things to when experimenting with an FPGA board or something.
But this is a nice near-term upgrade.

2026-04-21 07:25:05 UTC

- reply

After the Ceph node migration I now have only 11 active 10G SFP interfaces on the Nexus, one of which will be going away when I swap the VM server over to 40G.
Of the remaining 10, only 6 are real 10G loads and the other 4 are 1000baseX links to the 1G edge switches.
But once I deploy my own switching for the 1G edge, they'll have real 10/25G uplinks

2026-04-21 07:21:28 UTC

- reply

All wired up, now using 5x 40G + 1x 100G ports for the moment. Office workstation will get bumped from 40 to 100 at some point so then I'll have two free 40G ports, one of which will eventually get hooked to the VM server.
Plus three free 40G ports on the 10G switch.
Also I picked a hostname, it's argon2
https://files.ioc.exchange/iocexchange/media_attachments/files/116/441/480/998/810/783/original/5a71fe4ac4c7006d.jpg

2026-04-21 05:52:40 UTC

- reply

On this switch, trunks default to *no* vlans allowed. You have to do "switchport trunk allowed vlan all" before it will pass any traffic.