nirik :fedora: :redhat:

Sysadmin shouting at clouds. #fedora #redhat

Public Key

npub1hm6w3hk9x5jj25ljq68ldluukgqr29fq8p83u2ut7ysvyd9se0vsuqedjd

NIP-05 Address

nirik@fosstodon-org.mostr.pub

Profile Code

nprofile1qqstaa8gmmzn2ff920eqdrlkl7wtyqp4z5srsnc79w9lzgxzxjcvhkgpz3mhxue69uhhyetvv9ujuerpd46hxtnfduqs6amnwvaz7tmwdaejumr0ds3625tm

Publishing to

relay.damus.io

nos.lol

Author Public Key

npub1hm6w3hk9x5jj25ljq68ldluukgqr29fq8p83u2ut7ysvyd9se0vsuqedjd

Show more details

Last Notes

2026-07-12 00:35:02 UTC

Why don't all #zigbee devices have a 'recheck battery and report' call? It's anoying to replace a battery in something and you can't tell for days when it gets around to reporting battery again. ;(
#homeassistant

2026-07-03 21:07:10 UTC

- reply

@nprofile…pmjm mine hasn't shown up yet. ;( Wonder if I should try and followup...

2026-06-01 01:49:38 UTC

- reply

@nprofile…n635 It's a beef roast... want it to be nice and fall aparty. :) Getting hungry tho, so we will see how long I leave it.

2026-06-01 00:42:53 UTC

More yard work, more planting things.
Waiting for the meat to be done in the smoker.
Time for a cider... it's kinda dry.
https://cdn.fosstodon.org/media_attachments/files/116/672/063/931/989/597/original/3a72c21623f1b3b8.png
https://cdn.fosstodon.org/media_attachments/files/116/672/069/559/257/351/original/c562844a0872159b.jpg

2026-05-30 23:44:01 UTC

Spent the afternoon in the big blue room.
Had a helper ( #caturday ).
So glad this garden soil I got is for "in ground use" (as opposed to what? in water? sky? )
Got a bunch of weeds pulled up and zuchini and squashes planted.
https://cdn.fosstodon.org/media_attachments/files/116/666/167/620/732/876/original/d0badaa14f182927.jpg
https://cdn.fosstodon.org/media_attachments/files/116/666/175/066/003/483/original/8800fcffb26ae5c1.jpg

2026-04-25 22:44:58 UTC

The botanical gardens near hilo has some flowers that do not seem real.
https://cdn.fosstodon.org/media_attachments/files/116/467/764/880/331/224/original/b715f9bccd997758.jpg

2026-03-28 18:24:43 UTC

I have ~60W of load thats showing up as 'untracked consumption' on my #homeassistant energy dashboard. I know I should be happy everything else is known, but that last 60W is bugging me. ;)

2026-03-21 17:46:47 UTC

ok, another saturday, another weekly recap. ;)
Do let me know if anyone is reading these or finds them interesting.
This week we have: secureboot signing work, openh264 builds, openshift upgrades and mass update/reboot cycle next week. Enjoy.
https://www.scrye.com/blogs/nirik/posts/2026/03/21/misc-fedora-bits-third-week-of-march-2026/

2026-03-05 02:38:06 UTC

Somehow got behind this week. ;( In my #fedora corner:
Monday:
* Some meetings.
* Bunch of tweaking/testing with secureboot signing.
* Renewed some vpn certs
Tuesday:
* Heavy meeting day. Much meeting.
* Got drac license for a server finally sorted and submitted pr to do initial config on it. (1TB memory, that was a good investment :)
Today:
* Some smaller tickets
* Some more attempts to deploy fix for .crate downloading
* Gignormous amount of emails
* Started zabbix dash discussion.

2026-01-22 18:23:14 UTC

So, looks like the AI scrapers have moved on to... #fedora download servers. ;(
Working on putting them behind anubis as a first step to see if that mitigates or if they are type2scrapers (botnet from captive browsers).
https://forge.fedoraproject.org/infra/tickets/issues/13075

2026-01-17 18:21:34 UTC

Gotta love open source projects that have a release with the changelog of "bug fix" and you have to test it to see if it fixes _your_ bug.

2025-11-16 02:02:04 UTC

Some #homeassistant fun today.
Our "smart" shower never rejoined the wireless network after the last power outage. It's a "u by moen" one.
Reset it by:
* hold down down temp button
* select reset
* select wifi
* join it's wifi on your phone
* run app on phone and select reset wifi
* power cycle it and curse until it takes
* press and hold up temp for the homekit code
* delete in homeassistant, add the new one it detects (it seems to change HK code on reset).
Fun times.

2025-11-15 17:41:07 UTC

Finally another weekly recap (but it's been a few weeks) from my #fedora infra corner.
This time:
tcp timeouts fixed! 🎊
a bunch of outages lately. 😢
power10 saga finally over. :blobcatcheer:
rdc-cc to rdu3 datacenter move news.
https://www.scrye.com/blogs/nirik/posts/2025/11/15/infra-weeksly-recap-early-november-2025/

2025-11-15 02:38:11 UTC

Today in infra #fedora land:
* kojipkgs servers hit a super high load (~900-1000). Tracked the issue to bunch of requests for ostree directory listings that have each 65k ish files in them, which apache has to stat each one. Moved these directories to be cached by varnish. Hopefully that will prevent the issue.
* Reassigned a bunch of packages with non human poc ( https://pagure.io/releng/issue/13079 )
* Fixed the f43 nightly cloud build tagging ( https://pagure.io/releng/issue/13079 )
<1/N>

2025-11-14 21:48:49 UTC

If you have finished investigating something, can you say that you troubleshot it?

2025-10-24 17:29:05 UTC

- reply

@nprofile…aa6g It's got a . in in... so it's one point six seven seven.
But still, 1 and 2/3rds inch per hour is... a LOT.

2025-10-24 17:24:13 UTC

Not the same. :)
The piezo in the WS90 is just reading really really high over the WH40. The WH40 seems much more correct.
https://cdn.fosstodon.org/media_attachments/files/115/430/293/445/354/046/original/707032430e9c36f0.png

2025-10-18 19:49:56 UTC

Saturday weekly recap!
This week: The timeout issue continues to frustrate me and take up all my time, please test the f43 release candidate, more power for copr, small datacenter move note:
https://www.scrye.com/blogs/nirik/posts/2025/10/18/infra-weekly-recap-mid-october-2025/
#fedora

2025-10-10 21:38:35 UTC

Got behind posting here this week, but:
Tuesday, Wed, Thursday, Today:
* Working on a really weird issue where sometimes we gets timed out requests to some backend servers. Still haven't found why, but I did get haproxy to largely paper over the issue by retrying. This affects kojipkgs and src.fedoraproject.org.
* Some work on setting up 3 more power9 machines for copr hypervisors in a new isolated network in rdu3. Adjusting network acls to get a install to work now.
* Usual pile of meetings.

2025-08-14 01:05:24 UTC

Today in my #fedora infra land:
* Looked at a report about our dns server sending... dns replies. I'm kinda baffled.
* Dropped one buildvm-a64 instance to keep a vhost from OOMs rarely.
* Upgraded our openshift clusters.
(upgrade, watch, done)
* Poked some more at the packages.fedoraproject.org app, got it to not error, but it also didn't seem to update correctly. ;(
* Processed a bunch of emails/requests/tickets/issues.
PSA: If you are having a issue, please please tell me the actual error!

2025-08-13 00:54:27 UTC

Today in my little corner of #Fedora
* Helped out with mass branching fedora 43. Overall pretty smooth this time. Only outstanding issue is the f43 metalink isn't working ( https://pagure.io/fedora-infrastructure/issue/12720 ).
We have rawhide and f43 branched composes done.
* Grew storage on packages, hopefully it will be working now.
* Got our vHMC (virtual Hardware management console) vm up and poked at it a bunch. Presented some options to networking folk on how we could get it working. Java is so great.

2025-08-12 00:35:16 UTC

Today in my #fedora infra land:
* Sorted some failed updates composes.
* Reviewed a bunch of PR's for f43 branching tomorrow.
* Started eln resigning with the fedora-44 key
* PR to drop oz/ImageFactory from our builders now that we don't use it anymore.
* Fixed perms on meetbot logs so the 'teams' linking process could work again.
* Dropped the old pdc database that was taking up many 100's of GB.
* Looked at docs for vHMC install... tomorrow.
* Planned to take thu/fri/monday off.

2025-08-06 00:06:58 UTC

- reply

@nprofile…5rql Enjoy! I'll keep the fires burning... oh wait, that might not have been the right way to put things?

2025-07-28 22:41:57 UTC

- reply

@nprofile…5rql It turned out to be in my vpn config from long ago sometime. ;( So, no actual openvpn or NetworkManager-openvpn bug.

2025-07-28 21:35:03 UTC

So far, only brokenness was openvpn. It was trying to use the ca-bundle that was dropped. ;(

2025-07-20 16:29:06 UTC

- reply

@nprofile…5rql but but but... lasers!

2025-07-19 22:53:56 UTC

Finally got to setting up my ratgdo that I got before flock.
The actual setup was super easy. Power on, up on bluetooth, HA says 'setup?' Update firmware. Done on software side.
Wiring was also easy.
Mounting it was a super pain. I got a magnetic mount, but it was useless. Still haven't found a place where the laser and vehicle sensor work yet. Will ponder.
Fun facts: ratgdo stands for 'rage against the garage door opener' and my garage has been opened 3,067 times.

2025-07-19 01:13:21 UTC

- reply

@nprofile…gkqt you mean bourbon isn't dinner by itself?

2025-07-16 00:31:00 UTC

Monday in my #fedora corner:
* Restarted signing server when it went out to lunch
* Reenabled backups and ran initial set over the new machines in the new dc.
* Sorted most of the old dc hardware that was shipped to the new dc. (3 left)
* Got the fcos ppc64le builder back online
* Got the kernel qa machine back online.
* Some prep work for ipv6 enablement
* Started setting up iscsi for ppc builders, hit a problem and asked storage folks to look.

2025-07-14 22:40:36 UTC

Just booked hotel / got ticket for fossy.
In previous years I could never go because flock was around the same time. Since flock moved to June this year I can finally go.
Any other folks going?
https://2025.fossy.us/
#sfconservency

2025-07-12 21:00:52 UTC

So, yesterday both my main system upses alerted and then refused to power back on with a overload message.
Finally pulled things, repowered them and they came up, but with 60% battery and not charging.
Very likely they have the 'cyberpower conductive yellow glue of death' problem. ;(
Ordered some new upses. Once they are in I'll open these and see...

2025-07-10 01:03:33 UTC

Today in my #fedora world:
* fedora.im non federation was caused by... drum roll... DNS! We hadn't updated it and it was pointing to the old DC, so the well-known files were not resolving, which matrix uses to find out servers. ;( Fixed.
* Fought with stuck f41 updates-testing compose. Still haven't solved it yet. ;(
* cleaned up a bunch of things in nagios
* Deployed a few smaller machines we hadn't yet.
* Got ready a massive DNS update, will push tomorrow.
* Fixed some forks on src.fp.o

2025-07-05 22:00:17 UTC

A post #fedora datacenter move post:
https://www.scrye.com/blogs/nirik//posts/2025/07/05/whew-datacenter-move-mostly-over/index.html
whew.

2025-06-25 23:39:54 UTC

- reply

@nprofile…5xqk I think it's fixed? try now?

2025-06-25 23:36:20 UTC

- reply

@nprofile…5xqk huh, ok will look.

2025-06-19 01:32:16 UTC

Today in my world:
* Some meetings.
* Sorted out vlans for new power10 boxes. Fixed dhcp config for it, it gets an ip, but doesn't pxe boot. :(
* Worked with @nprofile…972c to setup a new power10 fedora coreos builder. Went pretty easily. Hopefully it's all working, more testing to be done with it.
* Tried to sort 25G interfaces on arm machines. Looks like it might be a cable/SFP/switch issue in the end. Asked networking folks to look.
* Updated new openshift cluster. Went just fine.
* Mor Planning

2025-06-16 18:52:50 UTC

Hum... the fedora contributor survey lists "Diaspora" as a social media.
Anyone still using it?

2025-06-15 20:07:58 UTC

- reply

@nprofile…5rql This confused me completely at first.... "why doesn't this cap come off?"

2025-06-09 22:14:08 UTC

Hot off the presses, my flock to fedora recap. Was a great time!
https://www.scrye.com/blogs/nirik/posts/2025/06/09/flock-to-fedora-2025/index.html
#fedora #flock2025

2025-06-09 03:11:16 UTC

Made it home from flock earlier today.
What a great time! I'm sure going to need to catch up on sleep though.

2025-05-22 00:26:33 UTC

- reply

@nprofile…5rql Yes, this problem affects me too, and it's anoying. ;( But I don't think the fedora theme causes it? Just koji in general + dark ? I'll take a look at that issue tho, thanks!

2025-05-22 00:23:23 UTC

Monday in my #fedora world:
* Built new koji with our theme / compose patches for updates later in the week.
* manually installed another machine in datacenter to confirm HSMs really being there.
* Discussed a bunch of signing HSM plans
* Blocked some spam users in various places.
* More DC move prep work for when network appears.

2025-05-07 21:36:33 UTC

Setup 2 new identical servers. The only setup difference was one I setup with the onboard hardware raid, the other with linux software raid.
I tossed a identical fio job at both.
The hardware raid setup finished it in 24m09s.
The software raid setup finished in 20m22s.

2025-05-04 23:21:39 UTC

Finally was able to get time/energy to take apart the entertainment center, get out the ladder, take apart the heatpump unit thats at the top of our great room and replace (with a longer cable this time!) the cable attaching my esp32 to the diag port.
Whew, it's all working now. Pity I can't get power readings off these, but just being able to control them without the stupid remote and have historical data is a big win.
#HomeAssistant

2025-04-19 01:30:27 UTC

Today in my #fedora world:
* Restarted pki_tomcatd when it was in a weird state on ipa servers.
* Cleaned up a build that was messed up by the koji DDoS eariler in the week. It just needed to be rebuilt.
* Sheparded a bunch of packages through signing to get the queue processed quicker than it would have
* Pushed a few post freeze PR's that were still pending.
* Helped roll out nftables to all the prod builds. So far so good.
* Setup some hoses to trees so I could automate watering them.

2025-04-15 22:46:00 UTC

Impressive. Picked up a pair of powerline adapters. Plugged in one in my computer closet and another in our pumphouse. Just works. Now I have network down there and can put sensors in it. :)
#HomeAssistant

2025-04-11 01:05:17 UTC

Today in #fedora infra land:
* Worked on a DC move post (out soon I hope) with details I shared in my last blog post.
* F42 go/nogo meeting. We are surprisingly 'go' on a first rc. :) Thats great news, although there's a bug with initial-setup and Xfce live's that I didn't start looking at until too late. Sorry about that. Hopefully we can docuement a good workaround.
* went to the dentist and got a new crown. My face is all numb now.
* Merged some nftables cleanup pr's from @nprofile…kllr

2025-04-11 01:02:20 UTC

Wed in #fedora infra land:
* Talked with CoreOS and openqa about moving to new datacenter early. Might work.
* Did some testing in staging with adding a index to the silly mailman3 bounceevents table. Then did it in production. Wow. From 100% i/o all the time to still some high spikes, but not nearly pegged. It's amazing what a index will do. Still writing the 50G table is anoying tho.
* Cleaned up some bouncing list users.
* Did some reviews on docs/pr's.

2025-04-04 01:06:19 UTC

Today in my #fedora corner:
* Looked at why so few dnf-automatic emails. Turns out @nprofile…5rql patch to not send emails on no updates finally landed. Hurray!
* Helped @nprofile…0s9k land some epel10.1 changes to help handle upgrades between minors. (He did all the work, I just watched)
* Blocked several projects on pagure.io that crawlers really really wanted to crawl. This dramatically improved things. Probibly won't last, but nice to see load so low again.
<next toot for some datacenter move stuff>

2025-03-14 00:47:39 UTC

Today in my #fedora corner:
* Longish (but not a record) go/no-go for Fedora 42 Beta. We are a GO! Release on tuesday.
* Renewed some openshift api certs.
* added some folks to sysadmin-dns and went over how things worked.
* Filed a few PR's on ansible for post freeze and some docs pr's.
* Sorted out ima der files for f43/f44, filed a fedora-repos pr and a docs pr to do this at branching.
* Poked a bunch of tickets, closed a few.
* Updated DC move ticket, target is looking like June now.

2025-03-13 01:10:32 UTC

Today was busy busy in my #fedora corner.
* Yesterday I was off, so double catch up on emails and meetings.
* More blocking AI bots. Discussions about blocking. Unblocking a legit IP that got blocked again. Blocked alibaba cloud.
* Worked with onsite person at datacenter to replace a failed power9 drive, track down a missing machine (it was there, but never had been racked). Fixed serial numbers on some other machines.
* Fixed splunk endpoint per splunk team
<next toot for fun part>

2025-03-09 01:48:38 UTC

ok, got my stupid smart shower added to home assistant. Now I guess I can remotely turn on the shower in case... no, drawing a blank on a use case here.

2025-02-28 02:51:49 UTC

- reply

@nprofile…5xqk won't power on at all. Just dead. ;(

2025-02-28 02:50:14 UTC

Today in my #fedora world:
* kojipkgs fix seems working. hurray!
* Short bodhi outage caused by database load spike (which stopped as I was looking)
* Spent some time hunting hardware in datacenter. There is a few things that aren't accounted for correctly still, but progress made.
* Worked on auth for riscv seconday builders. Have a keytab setup thats working, but hostnames are not very flexable.
* Made 20 calls about getting someone to fix our dishwasher. Found a place that can come in 2 weeks

2025-02-16 17:36:48 UTC

- reply

@nprofile…5rql I have no windows. ;) I can look if it works in wine or something I guess...

2025-02-16 17:34:01 UTC

Nice to see folks pulling their ebooks off amazon to have and use locally.
I wish I could do that... but I can't since I removed my old, broken kindle. I don't want to buy one just to be able to download my books.
Oh well. I'll just keep buying elsewhere...

2025-01-16 23:04:55 UTC

And on to wed (yesterday) in my corner of #fedora land:
* forgejo meeting. Lots of ideas and planning beginnings. Hope it will be great!
* Applied a bunch of rsync updates.
* A small bit more work on riscv koji hub ansible def
* Fixed some failed updates composes
* Fixed some fallout from a bunch of ansible lint cleanups.

2025-01-14 19:58:30 UTC

QE job open on my team!
https://redhat.wd5.myworkdayjobs.com/en-US/jobs/job/Waterford-City/Software-Quality-Engineer---Fedora-Linux_R-045047-1?q=fedora
I think it's a great place to be...

2025-01-11 19:07:26 UTC

Huh. gitlab tells me that my ssh key(s) are expiring. Thats somewhat anoying. I suppose it forces people to use a seperate ssh keyset for gitlab.

2024-12-23 18:33:26 UTC

Moved my blog after 21 years.
Finally just moved to a static site from wordpress.
https://www.scrye.com/blogs/nirik/posts/2024/12/22/hello-from-nikola/
Perhaps this will make me blog more.

2024-12-10 02:37:52 UTC

Monday in #fedora infra land:
* Todays rawhide compose FINISHED (100% green. 🎊 )
* Setup a 'ci' channel in koji with only 2 s390x builders to prevent ci jobs from swamping things.
* Talked to mainframe folks and they tweaked stuff to get our lpar moe performant. Also, they are going to add more memory and disk (but not cpu).
* Looked into the weird /dev/pts issue with image builds ( https://github.com/OSInside/kiwi/pull/2692 )
* Pushed fixed koji pkg to epel10
* Setup a riscv db for secondary koji. hub next.

2024-12-09 21:52:59 UTC

FESCo (fedora engineering steering comittee) elections are open!
I'm of course running again...just can't get enough. 😁
If you are a fedora contributor, please vote:
https://elections.fedoraproject.org/
you can even (optionally) claim a badge:
https://badges.fedoraproject.org/badge/i-voted:-fedora-41

2024-12-09 16:19:46 UTC

Woah! Our work here is done:
https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20241209.n.0/STATUS
This FINISHED means the rawhide compose completely and every single artifact correctly built. This almost never happens, there's always some dep issue in something or a compose problem with something or whatever.
#fedora #rawhide

2024-12-07 17:37:20 UTC

Friday in my #fedora land corner:
* Odd rawhide compose failures. Seems like it might be a kiwi chroot build is somehow unmounting the hosts /dev/pts mount. Very odd.
* Setup some bits to get zuul runs on pungi-fedora pr's (but the hard part was submitted by @nprofile…wgg8 )
* Updated a bunch of random things to f41 from f40.
* Some more FY25 hardware planning stuff.
* Upgraded bodhi to 8.3.0, and f41, and it didn't go great. Had to move the openshift app back to f40 and had to <next toot>

2024-12-06 19:35:28 UTC

'ansible -m reboot host' is so much nicer than just logging in and rebooting it.
It checks and tells you when it's back up and answering ssh again, no pinging or going off to do something else and forgetting it.

2024-12-06 02:17:07 UTC

Today in #fedora land:
* Looked at rawhide compose failure, rebooted a aarch64 builder that was in a weird state and fired a new compose
* Poked more at the staging openshift cluster upgrade being stuck. Asked smarter people than I about it, but not much reply yet. ;(
* Upgraded compose-x86-01 to f41.
* Did some looking to see what I would need to dig out of git history to setup the secondary riscv koji hub in ansible.
* Caught up on a bunch of emails/pr's/issues

2024-12-05 23:18:59 UTC

Tuesday in my #fedora corner:
* Meetings. Lots of meetings.
* Revoked and reissued a bunch of openvpn certs.
* debugged our sudo rules. Turns out the order of files sss in nsswitch.conf matters a lot. ;)
* Resumed a failed f41-updates-testing compose.

2024-11-29 19:59:00 UTC

- reply

@nprofile…5xqk unlikely. ;)

2024-11-29 19:31:03 UTC

What a non relaxing morning. For some reason the audio part of the HDMI out on my media pc stopped working after I updated it to Fedora 41.
Tried a ton of things with no luck.
Oh well, will look at it again later with fresh eyes.

2024-11-27 00:46:12 UTC

- reply

@nprofile…9k7h oof. communication is hard. ;(
We should try and have a discussion when we move to a new forge what terms are more acceptable to everyone.

2024-11-26 18:54:05 UTC

- reply

@nprofile…ghct yeah, but note that that is in dnf4. I don't think there's a dnf5 version yet (if you happen to be on f41+), but yeah, should still work ok for a while...

2024-11-26 18:37:22 UTC

- reply

@nprofile…ghct python3-tracer?
that has the 'tracer' command that tells you what services need restarting and if you need to reboot.

2024-11-26 18:35:24 UTC

multi-cultural mid morning snack:
"English" muffin
"Canadian" bacon
"American" cheese

2024-11-23 20:56:22 UTC

- reply

@nprofile…hhvx I discovered 'sort -h' a few weeks ago. Well worth the Premium version!

2024-11-23 18:22:19 UTC

Saturday bonus toot from #fedora infra land:
* Sorted out our libvirt rawhide compose issues I think.
First, the nftables backend wasn't working right for nat. Just moved back to iptables until we switch.
Secondly, was of course partially my fault. ;) I had restarted libvirtd by reflex after some changes, but that stopped all the sockets of the split out libvirt. Ended up just rebooting thos builders to get them back to normal.
New rawhide compose running. 🤞

2024-11-23 18:17:41 UTC

Friday in #fedora infra land:
* Internal tabletop exersize. Was fun (I was just an observer)
* Call with datacenter folks about a server with a bad power supply. Removing and inserting it and... it's all green again.
* log01 (our central log host) alerts on low disk space. Seems out staging poddlers app is spewing tracebacks to it. Scaled it down to 0 until others can fix it.
Compressed 250GB of logs
* Finally got pesign/secure-boot signing all happy.
* looked at rawhide failures, fixed a few

2024-11-23 18:14:27 UTC

Thursday in #fedora infra land:
* Fixed broken ImageFactory plugins on builders (the links break on upgrades where the python version changes)
* pesign/secure boot signing was broken. Turns out opensc decided that if you only have one smartcard, you don't need to know it's name. Worked around that in new pesign build. Then some perms problems to sort through.
* Ran a bunch of playbooks to make sure things were in a desired state.

2024-11-20 01:28:42 UTC

Oh, and also today: bomb cyclone.
Probibly will loose power tonight and have to fire up the generator.
https://www.accuweather.com/en/weather-forecasts/bomb-cyclone-with-atmospheric-river-to-blast-northwest-part-of-california/1715099

2024-11-19 03:31:41 UTC

Today in my #fedora corner:
* Monday. Very monday.
* Bunch of meetings with more meetings.
* Upgraded koji hub in staging to f41 and... it broke. Will need to investigate further.
* Updated/rebooted a bunch of staging hosts.
* Finally reinstalled that last rhel7 box and boy was it a rabbit hole. Was booting BIOS instead of uefi, but couldn't get the old drac to do a virtual image. No PXE at that site. Just re-installed it with bios boot for now. Bunch of other doom along the way.

2024-11-17 02:24:41 UTC

- reply

@nprofile…5xqk which flavor? I remember that stuff....

2024-11-15 21:20:54 UTC

- reply

@nprofile…3yyd Sure, signed repodata could help the case of a local sync of all the repo.
Sadly https is the anchor of trust for everything. Where did you get your distros media and gpg keys?
Note also that all fedora rpms are signed.
Anyhow, we did get pretty far along to enabling signed repodata, but when we last tried it dnf had some interface problems with it (erroring out without saying why, asking people to accept new keys that they already had, etc). We might try again soon. todo list long.

2024-11-15 18:24:41 UTC

- reply

@nprofile…3yyd no.
If you are using the default metalink, hostile mirrors cannot mess with system updates, no repodata signing needed.
The metalink (which you get from the project mirrorlists) includes the checksum for the repomd.xml file. A mirror that has changed that will not validate and be skipped. That file in turn has the checksums for all the other repodata including the packages, so if they are not exactly right they will also be rejected as invalid.

2024-11-15 17:44:36 UTC

- reply

@nprofile…n2wp yes! But what did she do today?

2024-11-15 00:52:10 UTC

Thursday in #fedora infra land:
* Got up to a lists not working issue. Turned out our mailman instance applied security updates and got a RHEL9.5 postfix update and it was in a weird state. Restarted it and the lists traffic started to flow. Also, a LOT of double bounces for admins. ;( I cleaned up about 10k of them from the queue.
* Reinstalled some openqa aarch64 workers with f41.
* Caught up on a ton of old emails I had marked to reply to. Only 13 left marked to deal with.

2024-11-15 00:49:49 UTC

Wed in #fedora infra:
* Fixed some failed updates pushes
* Cleaned up some logs on a copr-hypervisor that alerted about low disk
* More meetings
* Found out nested virt was messed up on most (but not all) ppc64le builders. Upgraded hypervisors to f41 and fixed nested-virt on all the guest manually. I couldn't find a way to do it via virt-install or the like. Need to file a libvirt bug.
Hopefully they are some better for normal builds, but image builds should be a LOT faster.

2024-11-15 00:47:30 UTC

Tuesday in my little corner of #fedora infra land:
* Pile of meetings, but almost all of them useful and interesting. :)
* Some tweaking of src.fedoraproject.org. Folks are seeing sporaic 503's from it. https://pagure.io/fedora-infrastructure/issue/12260 if you see this and want to help us isolate it.
* Rebooted a power9 machine I upgraded to f41 the other day.
* Usual issues, emails, matrix discussions, etc.

2024-11-08 20:22:23 UTC

Thursday in #fedora infra land:
* A DDOS against our kojipkgs machines. 2 ip's hitting about 500k times in a very short period. Blocked them.
* Worked with @npub14d3…7npp to get epel10 stuff setup before release.
* Bunch of discussion about a corner case in epel retirements (when a epel package goes into rhel and other epel packages build against it). I'm for dropping the automation there because it's almost never used and make it something to manually fix at rhel minor's.
* Updated tickets

2024-11-08 20:19:42 UTC

Wed in #fedora land:
* woke up at 4:04 am. Thought to myself "sleep not found?"
* Reinstalled noc01 and secondary01 with rhel9 (from 8)
* Blocked some spam users
* bunch of meetings.

2024-11-08 20:18:28 UTC

Haven't been updating much this week for obvious reasons, but here goes:
Tuesday in #fedora infra land:
* Upgraded some f39 machines to f41.
* Deployed Bodhi 8.2.0 ( Thanks to mattia and @npub12h3…9qyg for all the work on it)
* Moved some config around to add openqa workers from old koji builders.

2024-11-02 17:03:16 UTC

Friday in #fedora land:
* Updated our openshift clusters. Went very smoothly.
* Blocked a bunch more nets on pagure.io. AI scraping fun. ;(
* Pushed a patch to koji to set use_buildroot_repo back to false for kiwi builds. Need to talk to upstream about it.
* bvmhost-p09-03 having problems again. Rebooted and will watch it more.

2024-11-01 19:10:43 UTC

Thursday in #fedora land:
* pagure.io outage to add some disk space. It was getting low. Added 500G so hopefully that will keep it for a while.
* Blocked some more ip's/networks on pagure.io that were being abusive.
* Setup some bucket lifecycle for fedora coreos folks.
* another server with a dead disk. ;( This time a power9 box. Is it something in the water?

2024-11-01 18:34:31 UTC

- reply

@npub18ts…4xrx glad to hear it!

2024-11-01 18:26:16 UTC

Tuesday in my corner of #Fedora
* F41 release day!
Did not go super smoothly. We had some unsigned checksum files and some aarch64 images in the wrong place. ;( Fixed those up quickly and the rest went pretty well. We had to regenerate the torrent files too, so if you are seeding please make sure you have the current ones.
* Strangely debuginfod went unresponsive and I power cycled it. Had some fs errors, so that needed sorting out.
* Bunch of reading and watching for issues and routing them.

2024-10-29 22:54:24 UTC

Mailed my ballot in the other day and just checked:
"Your ballot was received on October 28, 2024."

2024-10-29 18:51:31 UTC

- reply

@npub1jrp…xxnn Cool. Keep those bits flowing. ;)
And thanks again for setting up the micromirror network!

2024-10-29 18:43:32 UTC

Wiating to see what the #MicroMirror network thinks of #fedora 41 release. ;)
CC: @npub1jrp…xxnn

2024-10-29 18:33:12 UTC

Got behind again. :)
Monday in #fedora land:
* Looked into some sporadic 503's from src.fedoraproject.org, blocked some ip's, tweaked a few things, nothing stands out tho.
* Got atomic desktops refs sorted for release. Many thanks to @npub1cpm…uycf ( https://pagure.io/releng/issue/12420 )
* Fixed f41-updates-testing compose, then did a stable one to fix refs.
* Bunch of prep for release tuesday.

2024-10-22 00:32:22 UTC

Today (monday):
* More next year hardware planning stuff.
* Bunchomeetings
* Voted in the blocker bug meeting a bunch
* Removed an old ghost of odcs from builders. (old mount for volume thats no more)
* Fired off a 1.3 f41 compose after samyak made config/tagging changes.
* Caught up on a bunch of internal meeting recordings from last week.
* Did some test builds of Server guest images with kiwi.

2024-10-21 22:52:19 UTC

These Coke oreos don't taste too much to me like Coke, but they sure got the smell down pat.

2024-10-21 02:21:34 UTC

- reply

@npub13t5…qflp @npub1y3q…2weu Uh, I am often forgetfull these days, but I don't recall talking with you excellent gentlemen recently?