Codeberg Status

https://social.anoxinon.de/@codebergstatus

Service status announcements from the https://codeberg.org system administrators.

Replies to this account are not monitored. For more exciting news, make sure to follow Codeberg (npub1fh7…ee83). :)

Status Page: https://status.codeberg.eu/status/codeberg
Infrastructure Repos: https://codeberg.org/Codeberg-Infrastructure
Main Account (That You Likely Already Know Of): Codeberg (npub1fh7…ee83)

Public Key

npub1z0ase7phympjeflh9wqm6j3gyuwcftpu7udummcfnr49s5yf0p5qfmdh9p

NIP-05 Address

codebergstatus_at_social.anoxinon.de@momostr.pink

Profile Code

nprofile1qqsp87cvlqmjdsev5lmjhqdafg5zw8vy4s70wx7dauye36jc2zyhs6qpz3mhxue69uhhyetvv9ujuerpd46hxtnfduqs6amnwvaz7tmwdaejumr0dsx5mjep

Publishing to

relay.primal.net

relay.damus.io

nos.lol

Author Public Key

npub1z0ase7phympjeflh9wqm6j3gyuwcftpu7udummcfnr49s5yf0p5qfmdh9p

Show more details

Last Notes

2026-05-27 14:38:35 UTC

Good news: We have addressed recent SSH performance degradation by replacing linear parsing of the authorized_keys file with a custom AuthorizedKeys command.
This is what GitHub and GitLab have been doing for years, and we have grown to a size where this has become necessary for us as well.
The cause for the degradation was still abusive patterns, as connections without valid key take more resources (scanning the file to the end) than legitimate users (scanning is stopped after match).

2026-05-18 21:17:32 UTC

- reply

The main Forgejo instance is back, SSH also works again. We have no remaining maintenance for this evening, all services should be working again without any hiccups.
If you feel or see any problem, please report it to https://codeberg.org/Codeberg/Community/issues

2026-05-18 21:02:00 UTC

- reply

This reboot has now finalized, BIOS and iDRAC update was successful. We're now waiting on the Galera node to be fully synced to the cluster and then we'll move the main Forgejo instance back.

2026-05-18 20:25:08 UTC

- reply

We're starting the last reboot of the evening, the one that has the main Forgejo instance. We've moved this to another server temporarily, all data, queues and session were already stored high availability a while ago so this is luckily a trivial task. Unfortunately we forget to sync SSH host keys, that will stay down
This one has not been rebooted for 110(!) days. We'll also be updating the BIOS and iDRAC with this reboot, so this one will take a bit longer. But no services should be impacted.

2026-05-18 20:00:47 UTC

- reply

We noticed that during this reboot, CephFS barely responded to any FS I/O requests. This caused 504 errors to be seen on codeberg.org during this reboot; that was not our intention sorry for that.
The server we rebooted was the same that had "problems" where Ceph didn't allow it to go into maintenance mode: https://social.anoxinon.de/@codebergstatus/116523668707929156
We now have a feeling the problem is still not completely fixed and there's still a big dependency on this server for Ceph to work.
#note1pax…8rm6

2026-05-18 19:44:04 UTC

- reply

We'll be starting the reboot of the second server. This server only has services that are deployed with high availability, except the Woodpecker agent.

2026-05-18 19:18:37 UTC

- reply

This rebooted completed. We noticed that HAProxy didn't start on its own because it tried to read from Ceph before its mounted, we forget to add a mount dependency in the systemd service.
Unfortunately all services that uses the Galera cluster takes more time to start. We noticed for a while that when a Galera node joins the cluster it does a full SST instead of a incremental sync. Which is painfully slow.

2026-05-18 18:59:02 UTC

- reply

We'll be starting to reboot the first server. This server hosts Pages, Translate, Forgejo runner, e.V. forum and join.codeberg.org.

2026-05-18 18:48:21 UTC

- reply

To be clear, we mean to boot into a patched Linux kernel version. We have no intentions to migrate to a BSD distribution.

2026-05-18 18:28:33 UTC

You might've heard it by now: the Linux kernel is so insecure that AI can find many LPEs for them, so we'll be taking this evening to migrate to *BSD.
In all seriousness, although we've mitigated the currently known LPEs via blocking modules, seccomp and sysctl, we still want to run with a kernel that has patched it. We will be doing that this evening. That means that each server will be taken down to boot into the new kernel, and services will be temporarily inaccessible.

2026-05-08 17:07:58 UTC

It has been in the works for months, but the recent increase of downtime has accelerated work. We've moved from Caddy to HAProxy for codeberg.org traffic.
Starting over with a new reverse proxy has enabled us to make choices that enable us to set up HAProxy to scale with our increasing traffic. For example, traffic is split (when possible, decided via heuristics) over several instances to load balance expensive traffic.
We hope to share more details at a later time about this change.

2026-05-08 09:29:16 UTC

Please ignore.
Hi Matrix chat!

2026-05-05 19:41:21 UTC

- reply

Hi,
It's taking longer than expected. Ceph made a surprise move on us at the last moment by telling us our network configuration is maybe no longer up to date after we replaced servers (not today, but a while ago). This is preventing us from deploying new monitors to keep quorum. We're working on fixing this.
It's _simply_ preventing us from gracefully shutting down, no data is at risk.

2026-05-05 17:55:54 UTC

We'll be performing a reboot of all our servers this evening.
This means that our services (CI, pages, translate) will be temporarily offline. We invested a lot of effort in the last few months to be able to move Forgejo between servers easily and this should see minimal downtime.

2026-04-30 16:51:03 UTC

- reply

Hi all, sorry for the long wait.
We wanted to preemptively ensure that our servers were sufficiently shielded against https://copy.fail; we found no evidence of a compromise.
We waited for the greater community to react and provide some additional clarity on what is affected and what isn't (e.g. container engines), we carefully studied the LKML, and waited for upstreams to release official updates and guidance.

2026-04-29 22:18:43 UTC

- reply

We estimate that the recovery process of our CI services may potentially take multiple hours. We ask for your understanding as we are trying to reinforce our services' reliability.

2026-04-29 22:02:40 UTC

- reply

We are expanding our maintenance efforts to include our Forgejo Actions CI as well. Again, we apologize for the inconvenience. We'll try to bring both services back up as soon as possible.

2026-04-28 19:01:36 UTC

Our hosted Woodpecker CI/CD is going down for maintenance.
* Builds will be delayed.
* Web UI will be inaccessible for some minutes.
* Commits pushed during the downtime will not trigger builds.
We apologize for the inconvenience.

2026-04-02 07:16:21 UTC

We are aware of reports of Codeberg being down and are investigating. Thank you for your patience.

2026-03-05 01:30:46 UTC

- reply

We were able to restore IPv4 and IPv6 connectivity. Performance is still not optimal, and connections are occasionally dropping. We've taken measures to try and maintain uptime overnight (with a certain probability).
Routes protected by iocaine (crawler protection) may return 503 errors. Currently, it appears that these packets are also occasionally being lost.
It's 2:30am, and we require a good night's sleep before we can look into this further.

2026-03-04 22:09:08 UTC

Hi everyone! We are still fighting with an ongoing DDoS attack against our service on the network level. We do not yet have any ETA on recovery time, as we're still investigating different mitigation strategies.
As always, we are very grateful for your support and patience.

2026-01-15 18:20:59 UTC

- reply

Yesterday we didn't found the time to deploy, we will start doing another rebase of our patches on top of some last fixes and prepare for a deployment now. We are aiming to deploy at 20:00 CET (19:00 UTC)

2026-01-14 10:35:49 UTC

With Forgejo's scheduled v14 release for tomorrow, we will do our final tests of v14 tonight on our test instance.
If we can't find any issues with v14, we might deploy it today. We will let you know ahead of time when this happens, so you can prepare for 10-15 minutes of downtime.

2026-01-04 21:09:04 UTC

We're investigating an outage of our Ceph storage system. We'll hit you up once we know more.
Currently, most Codeberg services are not available.

2025-12-16 16:01:35 UTC

We are investigating performance issues on Codeberg.org.

2025-12-04 14:59:50 UTC

Welcome to our new dedicated account for service notifications surounding Codeberg's systems.
We hope there won't be much to share in the near future :)