Scheduled Forum update, upgrade, maintenance planned: 1800 UTC - May 28.

Code of conduct, suggestions, and information on forums.debian.net.
Post Reply
Message
Author
User avatar
donald
Debian Developer, Site Admin
Debian Developer, Site Admin
Posts: 1184
Joined: 2021-03-30 20:08
Has thanked: 196 times
Been thanked: 261 times

Scheduled Forum update, upgrade, maintenance planned: 1800 UTC - May 28.

#1 Post by donald »

Please take into consideration that the forums will undergo maintenance starting Tuesday, May 28 at 1800 UTC.


Timing:
This update addresses a few areas and will be done progressively starting on Tuesday, May 28th at 18:00 UTC.
The rain date for this maintenance plan will be Thursday, May 30 2024.

We plan to accomplish:
  • Administration:
    Checks on files of super super secret nature
    Detailed log reviews
Followed with the actual system updates/upgrades:
  • Forum Software:
    Software update to newest version
    Database cleanup
  • System software
    Operating system update and upgrade
  • Hardware:
    Nothing planned
- - - - -

Notes:

As indicated the steps for this maintenance window have a set order and may overlap a day or 2. While we do not anticipate any problems, users should be aware that some DNS changes will be made and forum availability may be impacted during those times.This part should resolve most of our email issues.

The database work will be focused on tweaking some memory settings and the clearing and cleaning of caches, no user accounts will be purged or affected. Some carefully audited older unanswered threads may be removed though that is rare and best left to a single dedicated task.

The system updates are very straight forward as they should be on any maintained system.

Due to unforeseen circumstances in our local gaze of Raccoons regarding an outbreak of rabies, mindfully now under control, Pringy our normal upgrade administrator will not be on hand to conduct these upgrades. We have as an administrative team elected to let an un-monitored AI instance perform the updates. Concern yourselves not, this AI instance is the same instance that flooded us with 27,000 bots from which we captured # 23,056. We have retrained this instance for the greater good. We are not responsible for PII leaked or VOIP calls from said instance. We wish you all the best of luck and hope that this paragraph in the least caught your attention.

- - - - -

This thread will also be used to provide updates after the process has finished, or to give updates if anything goes awry. We ask that everyone please keep all discussion here.
Typo perfectionish.


"The advice given above is all good, and just because a new message has appeared it does not mean that a problem has arisen, just that a new gremlin hiding in the hardware has been exposed." - FreewheelinFrank

memilanuk
Posts: 19
Joined: 2009-07-03 04:16
Has thanked: 1 time
Been thanked: 3 times

Re: Scheduled Forum update, upgrade, maintenance planned: 1800 UTC - May 28.

#2 Post by memilanuk »

Hopefully this will help speed up the forum a bit... it's been a tad laggy lately. And best wishes with the racoons ;)

User avatar
donald
Debian Developer, Site Admin
Debian Developer, Site Admin
Posts: 1184
Joined: 2021-03-30 20:08
Has thanked: 196 times
Been thanked: 261 times

Re: Scheduled Forum update, upgrade, maintenance planned: 1800 UTC - May 28.

#3 Post by donald »

memilanuk wrote: 2024-05-27 21:24 Hopefully this will help speed up the forum a bit... it's been a tad laggy lately. And best wishes with the racoons ;)
Great timing I was about to update this thread on some changes that we literally just made today.

The issue with the slowness has been a concerted bot attack from AI crawlers from Google, Microsoft, Amazon Web Services, ChatGPT, Claudebot, Huawei Cloud Services and so on.

I don't mind naming the services because they should be publicly shamed for their behaviour of blocking a public resource for their own selfish private capital gain. It is abhorrent behaviour.

We have been talking to some of those services, vendors, and startups, we have posted a please no AI training here thread, and we have been nice about it, but that is clearly not working and today we put additional measures into place.

In giving credit where credit is due, AWS is very responsive about this issue and they have done so on a few occasions with issues we have had. I'm not endorsing them for anything but I can say that of the myriad of companies that define themselves as open and consumer friendly yet offer no customer support email addresses, support addresses, or telephone numbers, AWS thus far seem to be the only one that you can talk to an actual person and that person will work toward the issues resolution. I do not presently use them for anything but if I had to choose an ExtraNet provider it would be someone concerned with their and my brand.

Anyhow, thank you to AWS, Debian System Administrators (ganneff), and my 5 new grey hairs who have all put measures in today to protect this public resource.

Enjoy the new speed!
Typo perfectionish.


"The advice given above is all good, and just because a new message has appeared it does not mean that a problem has arisen, just that a new gremlin hiding in the hardware has been exposed." - FreewheelinFrank

Augie77
Posts: 94
Joined: 2024-03-23 04:05
Has thanked: 19 times
Been thanked: 18 times

Re: Scheduled Forum update, upgrade, maintenance planned: 1800 UTC - May 28.

#4 Post by Augie77 »

This is good news. I have struggled for several weeks now getting access to the site. I even used Mullvad to get a German door, even then, it was a bit slow. I hope you are still able to keep the ProSilverColor-blinds theme, it is nice and easy on my eyes.

User avatar
donald
Debian Developer, Site Admin
Debian Developer, Site Admin
Posts: 1184
Joined: 2021-03-30 20:08
Has thanked: 196 times
Been thanked: 261 times

Re: Scheduled Forum update, upgrade, maintenance planned: 1800 UTC - May 28.

#5 Post by donald »

Augie77 wrote: 2024-05-28 00:08 This is good news. I have struggled for several weeks now getting access to the site. I even used Mullvad to get a German door, even then, it was a bit slow. I hope you are still able to keep the ProSilverColor-blinds theme, it is nice and easy on my eyes.
We hope the new changes work, but as always it is a current battle. While it may be in bad taste, I've changed my viewpoint to start naming the bad actors and continue praising the good ones in this never-ending battle.

No theme changes should happen with this update, but some times things do change between releases, from looking at the release notes it does not seem to indicate that any of our themes will change. In the past we have done a theme audit to drop/add themes, perhaps after the next update we can do the same and dedicate time to the broken ones that are in heavy use.
Typo perfectionish.


"The advice given above is all good, and just because a new message has appeared it does not mean that a problem has arisen, just that a new gremlin hiding in the hardware has been exposed." - FreewheelinFrank

User avatar
donald
Debian Developer, Site Admin
Debian Developer, Site Admin
Posts: 1184
Joined: 2021-03-30 20:08
Has thanked: 196 times
Been thanked: 261 times

Re: Scheduled Forum update, upgrade, maintenance planned: 1800 UTC - May 28.

#6 Post by donald »

Outside links blocked by prior policy are now available and working again. This affected only preview links on outside websites. Minor issue.
Typo perfectionish.


"The advice given above is all good, and just because a new message has appeared it does not mean that a problem has arisen, just that a new gremlin hiding in the hardware has been exposed." - FreewheelinFrank

steve_v
df -h | grep > 20TiB
df -h | grep > 20TiB
Posts: 1470
Joined: 2012-10-06 05:31
Location: /dev/chair
Has thanked: 94 times
Been thanked: 232 times

Re: Scheduled Forum update, upgrade, maintenance planned: 1800 UTC - May 28.

#7 Post by steve_v »

donald wrote: 2024-05-28 00:21While it may be in bad taste, I've changed my viewpoint to start naming the bad actors and continue praising the good ones in this never-ending battle.
IMO it's anything but bad taste. Please continue, with both the naming and the "additional measures".

I personally appreciate and applaud swift, non-negotiable bans for any subnet / IP pool seen to be running AI bots or crawling this board. Be as ruthless as you like and collateral be damned, this behaviour is abusive, exploitative and fundamentally corrosive to the open internet.

I never consented to any of my posts being used as training data, I never will, and I have zero interest in interacting with Artificial Idiots or reading their regurgitated drivel.
Once is happenstance. Twice is coincidence. Three times is enemy action. Four times is Official GNOME Policy.

Bulkley
Posts: 6396
Joined: 2006-02-11 18:35
Has thanked: 5 times
Been thanked: 43 times

Re: Scheduled Forum update, upgrade, maintenance planned: 1800 UTC - May 28.

#8 Post by Bulkley »

steve_v wrote: 2024-05-28 07:27 IMO it's anything but bad taste. Please continue, with both the naming and the "additional measures".

I personally appreciate and applaud swift, non-negotiable bans for any subnet / IP pool seen to be running AI bots or crawling this board. Be as ruthless as you like and collateral be damned, this behaviour is abusive, exploitative and fundamentally corrosive to the open internet.

I never consented to any of my posts being used as training data, I never will, and I have zero interest in interacting with Artificial Idiots or reading their regurgitated drivel.
Agreed.

memilanuk
Posts: 19
Joined: 2009-07-03 04:16
Has thanked: 1 time
Been thanked: 3 times

Re: Scheduled Forum update, upgrade, maintenance planned: 1800 UTC - May 28.

#9 Post by memilanuk »

The issue with the slowness has been a concerted bot attack from AI crawlers from Google, Microsoft, Amazon Web Services, ChatGPT, Claudebot, Huawei Cloud Services and so on.

I don't mind naming the services because they should be publicly shamed for their behaviour of blocking a public resource for their own selfish private capital gain. It is abhorrent behaviour.

We have been talking to some of those services, vendors, and startups, we have posted a please no AI training here thread, and we have been nice about it, but that is clearly not working and today we put additional measures into place.
I was aware of that sort of thing taking place on another forum (Linux Mint) a while back; I had no idea the various entities were still actively / overtly conducting that kind of shenanigans.

User avatar
donald
Debian Developer, Site Admin
Debian Developer, Site Admin
Posts: 1184
Joined: 2021-03-30 20:08
Has thanked: 196 times
Been thanked: 261 times

Re: Scheduled Forum update, upgrade, maintenance planned: 1800 UTC - May 28.

#10 Post by donald »

Updates/Upgrades completed on May 28 2024


Completed: System software. Moved to the most recent version of phpBB3.

Completed: Operating system update and upgrade. Upgraded from Debian 11 (bullseye) to Debian 12 (bookworm).

Skipped: Hardware. Nothing planned

Completed: Firewall checks and implementation

Completed: Log reviews: All is well.

Completed: Checks on files of super super secret nature. Ssshhhhhhhh! :)

Completed: Detailed log reviews - Nothing was amiss or concerning other than very large Apache log files from the bot network attacks. Will monitor and update our rotations if needed, but it may be best to hold the logs for a bit since I called out a few services and that data is the proof of the attacks overt intense curiosity.

Completed: Forum Software updated to the current version.

Skipped: Database cleanup. Considering our hardware, setup, and the database footprint in relation to the size of the forums, we skipped on data purges as they would not increase performance or add any real benefit for the process. When considering the search indices are also in the database the speed here is actually quite good. Why mess with something that is working better than initially thought?

Next Step: Networking. The above took a bit longer than anticipated, including writing all this and the notes to the other admins, so the email issue remains with us for an additional day. Will do it tomorrow (later today) as I have some other Debian DNS changes to make elsewhere. It makes sense to do everything there and here at the same time. Less errors that way. Sorry for the further inconvenience. In the meantime we have been manually approving some new user accounts which is kind of cool, some new users got an email 'interview' to join. :D To be fair I really like that approach instead of the blanket pass/fail the systems use especially as everywhere is now plagued by AI. ..... I'll stop ...

If anything is broken, or out of sorts please respond here. If everything is working ... please respond here, all feedback is appreciated.
Typo perfectionish.


"The advice given above is all good, and just because a new message has appeared it does not mean that a problem has arisen, just that a new gremlin hiding in the hardware has been exposed." - FreewheelinFrank

User avatar
donald
Debian Developer, Site Admin
Debian Developer, Site Admin
Posts: 1184
Joined: 2021-03-30 20:08
Has thanked: 196 times
Been thanked: 261 times

Re: Scheduled Forum update, upgrade, maintenance planned: 1800 UTC - May 28.

#11 Post by donald »

Number of entries: 932
Members:
52.233.106.44 BotService: "gptbot"
124.243.135.107 BotService: "huawei"
124.243.133.84 BotService: "huawei"
2.230.152.73 BotService: "gptbot"
124.243.134.238 BotService: "huawei"
52.233.106.126 BotService: "gptbot"
124.243.135.135 BotService: "huawei"
52.233.106.112 BotService: "gptbot"
124.243.149.22 BotService: "huawei"
124.243.148.74 BotService: "huawei"
52.230.152.230 BotService: "gptbot"
124.243.133.57 BotService: "huawei"
124.243.150.219 BotService: "huawei"
52.230.152.123 BotService: "gptbot"
52.233.106.125 BotService: "gptbot"
52.230.152.79 BotService: "gptbot"
124.243.149.239 BotService: "huawei"
47.76.99.127 BotService: "huawei"
52.230.152.78 BotService: "gptbot"
52.230.152.12 BotService: "gptbot"
52.230.152.180 BotService: "gptbot"
52.230.152.184 BotService: "gptbot"
124.243.144.175 BotService: "huawei"
52.230.152.198 BotService: "gptbot"
52.230.152.178 BotService: "gptbot"
3.19.31.73 BotService: "aweclaudeban.txt"


Since the reset. Ridiculous.
Typo perfectionish.


"The advice given above is all good, and just because a new message has appeared it does not mean that a problem has arisen, just that a new gremlin hiding in the hardware has been exposed." - FreewheelinFrank

Augie77
Posts: 94
Joined: 2024-03-23 04:05
Has thanked: 19 times
Been thanked: 18 times

Re: Scheduled Forum update, upgrade, maintenance planned: 1800 UTC - May 28.

#12 Post by Augie77 »

Thank you! All is working good from the East Coast, USA. :)

User avatar
nikobit
Posts: 142
Joined: 2009-02-08 19:40
Location: Moscow, Russia
Has thanked: 18 times
Contact:

Re: Scheduled Forum update, upgrade, maintenance planned: 1800 UTC - May 28.

#13 Post by nikobit »

All is sound and clear here in Mother Russia! Though some VPN got involved in the middle. Thank you for generous share of enormous efforts! Really appreciate that!
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  • OS: Debian 11 bullseye
  • Kernel: x86_64 Linux 5.10.0-29-amd64
  • DE: GNOME 3.38.5
  • CPU: Intel Core i7-4790 @ 8x 4GHz
  • GPU: llvmpipe (LLVM 11.0.1, 256 bits) / NVE4
  • RAM: 2384MiB / 15925MiB
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Aki
Global Moderator
Global Moderator
Posts: 3240
Joined: 2014-07-20 18:12
Location: Europe
Has thanked: 88 times
Been thanked: 429 times

Re: Scheduled Forum update, upgrade, maintenance planned: 1800 UTC - May 28.

#14 Post by Aki »

@donald
Thank you so much for all your efforts.
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org
⠈⠳⣄⠀

User avatar
donald
Debian Developer, Site Admin
Debian Developer, Site Admin
Posts: 1184
Joined: 2021-03-30 20:08
Has thanked: 196 times
Been thanked: 261 times

Re: Scheduled Forum update, upgrade, maintenance planned: 1800 UTC - May 28.

#15 Post by donald »

Updates so nice that we decided to do them again!

In sincerity my apologies for the system disruption, there were a few things going while I was already logged into the server, while there I finished this threads update/upgrade.


phpBB updated a new version just yesterday. Check out the release notes if you wish: https://www.phpbb.com/community/viewtop ... &t=2653732. Their update offers improvements around things we generally don't make use of heavily, there were some updates made to the prosilver theme (the theme all the other themes are based on) but not much to affect us. Since we had just updated, it made sense to apply that update as well.

@Augie77 Did the colorblind theme you use change? Please let me know ASAP and I will prioritize that theme fix or find you another one.

This update/upgrade:
  • Completed: Newest version of phpBB.
  • Completed:: Housekeeping: cleared forum caches, resynchronised user post counts, resynchronised topics, and user statistics, updated all extensions.
  • Completed: Deletion of old backups and un-needed files to reclaim 20GB of storage space.
  • Completed: Tweaked the search system as phpBB mentioned they had tweaked database connections. We checked to ensure nothing was out of sync.
  • Completed: Updated the Operating System again because I was still logged on and why not do such.
  • Completed: Feed the annoying cat who kept meowing for dinner while sitting next to its food bowl. Resolution was a tummy rub. This situation is on-going.
  • Pending: Fixed the email issue...waiting for a new registration or complaint to update that part.
Typo perfectionish.


"The advice given above is all good, and just because a new message has appeared it does not mean that a problem has arisen, just that a new gremlin hiding in the hardware has been exposed." - FreewheelinFrank

Bulkley
Posts: 6396
Joined: 2006-02-11 18:35
Has thanked: 5 times
Been thanked: 43 times

Re: Scheduled Forum update, upgrade, maintenance planned: 1800 UTC - May 28.

#16 Post by Bulkley »

Yes, thanks to you and all those who keep this forum running.

Augie77
Posts: 94
Joined: 2024-03-23 04:05
Has thanked: 19 times
Been thanked: 18 times

Re: Scheduled Forum update, upgrade, maintenance planned: 1800 UTC - May 28.

#17 Post by Augie77 »

donald wrote: 2024-06-01 23:11

@Augie77 Did the colorblind theme you use change? Please let me know ASAP and I will prioritize that theme fix or find you another one.

The theme is working great, just as it did before. Thank you very much!

User avatar
sunrat
Administrator
Administrator
Posts: 6771
Joined: 2006-08-29 09:12
Location: Melbourne, Australia
Has thanked: 122 times
Been thanked: 543 times

Re: Scheduled Forum update, upgrade, maintenance planned: 1800 UTC - May 28.

#18 Post by sunrat »

I'm heartened that you are treating your cat well @donald . Did you give the server a tummy rub as well? I'm certain that helps sometimes. :mrgreen:
“ computer users can be divided into 2 categories:
Those who have lost data
...and those who have not lost data YET ”
Remember to BACKUP!

User avatar
donald
Debian Developer, Site Admin
Debian Developer, Site Admin
Posts: 1184
Joined: 2021-03-30 20:08
Has thanked: 196 times
Been thanked: 261 times

Re: Scheduled Forum update, upgrade, maintenance planned: 1800 UTC - May 28.

#19 Post by donald »

During the update/upgrade process we were able to work with our wonky DNS setup to add SPF, DMARC, and DKIM records. Tests for the records was affirmative through several online query tools.

This morning after letting everything run for the past few days I registered a test account while running a journal over the logs, the email was received on gmail.
forums email subsystem wrote: Jun 03 19:58:41 forums.debian.net postfix/pickup[209912]: D7D09D: uid=33 from=<www-data>
Jun 03 19:58:41 forums.debian.net postfix/cleanup[216801]: D7D09D: message-id=<06140c3740dd373338767364f2@forums.debian.net>
Jun 03 19:58:41 forums.debian.net opendkim[35862]: 9409D7D09D: DKIM-Signature field added (s=forums.dkey01, d=forums.debian.net)
Jun 03 19:58:41 forums.debian.net postfix/qmgr[3887136]: 9409D7D09D: from=<www-data@forums.debian.net>, size=1600, nrcpt=1 (queue active)
Jun 03 19:58:41 forums.debian.net postfix/smtp[216803]: Untrusted TLS connection established to gmail-smtp-in.l.google.com[108.177.127.27]:25: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256
Jun 03 19:58:42 forums.debian.net postfix/smtp[216803]: D7D09D: to=<tt@gmail.com>, relay=gmail-smtp-in.l.google.com[108.177.127.27]:25, delay=0.7, delays=0.05/0.02/0.11/0.51, dsn=2.0.0, status=sent (250 2.0.0 OK 37522 23a62f3a-a687eeb3143si391819766b.675 - gsmtp)
Jun 03 19:58:42 forums.debian.net postfix/qmgr[3887136]: D7D09D: removed
Moments later I received an email for my new registration.
gmail account wrote: Welcome to "Debian User Forums"
Inbox

forums@forums.debian.net
1:58 PM (32 minutes ago)
to me

Welcome to Debian User Forums forums

Please keep this email for your records. Your account information is as follows:

----------------------------
Username: tstststs

Board URL: https://forums.debian.net
----------------------------

Please visit the following link in order to activate your account:


ucp.php?mode=activate&u=0&k=4m7x7cesurd ... jongptthg2

Your password has been securely stored in our database and cannot be
retrieved. In the event that it is forgotten, you will be able to reset it
using the email address associated with your account.

Thank you for registering.
Under the assumption that if gmail works the others should work, we can now safely consider this issue closed.
Typo perfectionish.


"The advice given above is all good, and just because a new message has appeared it does not mean that a problem has arisen, just that a new gremlin hiding in the hardware has been exposed." - FreewheelinFrank

User avatar
donald
Debian Developer, Site Admin
Debian Developer, Site Admin
Posts: 1184
Joined: 2021-03-30 20:08
Has thanked: 196 times
Been thanked: 261 times

Re: Scheduled Forum update, upgrade, maintenance planned: 1800 UTC - May 28.

#20 Post by donald »

Thread un-stuck from global announcements, remaining open for comments.

@sunrat meow :)
Typo perfectionish.


"The advice given above is all good, and just because a new message has appeared it does not mean that a problem has arisen, just that a new gremlin hiding in the hardware has been exposed." - FreewheelinFrank

Post Reply