My instance appears to have a problem - the first load of the network page takes so long that I've had to increase the timeout for php-fpm in my nginx config, or else get 504 errors.
It seems like later attempts to load the page are a bit faster, but I expect my users will have given up waiting long ago.
I've seen that others have had similar issues in the past, and there seems to be no solution.
iotop shows that even AFTER my profile page has loaded, a further 545MB of disk reads due to mysql goes on.
I'd noticed this sort of extreme disk activity going on before, albeit not as bad.
Is this a scalability issue in friendica? or is my node under attack?
Make sure you block those activity pub trolls, they will grind you to a halt.
Also if this is a new instance give it time to settle down. Friendica does alot of work in the first week or two to build out its understanding of other hosts, their accounts etc.
Increasing the workers is also helpful. And run it as a daemon.
I found a that shutting down mariadb each evening for backups and then restarting helped as well.
I think the workers were pushing the machine into swap when large qeries are caused by the frontend (I've now seen a query over 600MiB)
There is no way the system can handle >1 user if its going to cause 600MiB of disk reads every time they access the site!
Either something is wrong, or there is an incredibly inefficient query running somewhere, or a query is being run over an unexpectedly huge data set - but I have no way to know.
Whatever it is, it's crippling the machine - it seems to cripple it gradally enough that it doesnt immediately OOM (sounds long-query-ish to me), and it doesnt prevent more workers running - the machine is being killed by multiple threads seemingly eating ram whilst being "blocked" by a large running query.
Eventually the machine gets so bogged down that it can't serve anything. I can still log in (just - last time it had a load average of >30), and if I kill / restart mariadb, it will carry on, but it seems to be living right on the wire - a good push will cause it to start losing the battle and bog down again.
I like to go for an analytic approach. First please let us trace the requests. There we can quickly distinguish if the delays are possibly caused by a lack of available /Apache/Nginx/PHP connections or if the database is the bottleneck.
Depending on the result, we then can change some parameters.
I had this problem too. But even banning those spam domains and cleaning the database from their entries didn't have the effect that my server worked fine again.
The load stayed high and the responsing time was bad.
So I'm sure that in my case the change to another server with dumping and restoring the datebase was part of the problem.
Please have got a look at the requests, that your system receives. Switch the log level to "debug" and then run that command: tail -f friendica.log | grep -a Request.*'"runFrontend"' (Of course you have to change it to the name of your log file)
With that command you will see incoming requests. "Request received" tells you, that the request just arrived, and both "Request processed sucessfully" and "Request processed with exception" tells you, that the processing is done. BTW: Don't be confused by that "with exception", if the response code is in the 200 range, then everything is fine.
With these log entries you will see, if there are delays before the processing starts or during the processing. Depending on that, we should see, what to do then.
You have to trace individual requests. For example you can filter for network requests. Then you can see, when you sent your request via browser, when it appeared in the system and when it has been processed. We need that data to really see the origin.
Okay. When you get these 504s, then please perform this command: tail -f friendica.log | grep -a Request.*"GET /network".*'"runFrontend"' This will show you all the network requests. You will see the first entry, once the system received the request. When you don't receive an entry there (or only sometimes), then please consult your apache/nginx configuration and increase the number of possible connections or possible php processes.
Filter for example this way: tail -f friendica.log | grep -a 'Request processed sucessfully'.*'"runFrontend"' There you can see the successful frontend requests and their performance (in the field "duration"). Perform some frontend requests on the network page and see if the duration is high (multiple seconds) or not.
I've not done much - I've increased the php-fpm timeout in nginx so that some of the longer requests complete, bt this is looking like a database issue.
The VPS i'm running friendica on is essentially idle otherwise, has 4GiB RAM and two decent CPU cores.
Requests into the server aren't taking up much of its time, bmon shows ~4-15PPS a burst or two of 50-300PPS every 60 seconds or so. Peak bandwidth is about 2MB/s typical is more like 10-100K/s
Last night, mariadb crashed - I had increased the size of innodb_buffer_pool_size from 2 to 16G (yes, I know this is > RAM+SWAP). I reduced it down to 4 again, and it's seemingly ploughed through the worker queue as of this morning, but the whole site is still VERY slow.
Right now, from an adminn point of view, I see the biggest problem in friendica is that there is literally ZERO ability to introspect its behaviour.
I can look at the "worker queue", but there is no way to know what any of that means short of dismantling the code, which I dont have time to do.
I can't search the worker queue for issues in any meaningful way, and even if I could there is no mechanism to inspect / suspend / pause / cancel entries in the worker queue.
If I could suspend the entire queue, and selectively "gate through" certain entries, it'd be enormously helpful.
Lastly - of the "known" threats out there like activitypub-troll, is there a list of workarounds or a blocklist? HOW can we identify these malicious sites and keep our instances safe?
What is an (honest) minimum spec for a friendica node? I have to admit, whilst I was expecting a largish storage requirement, the RAM requirements are very mch higher than anticipated.
Yes. As described it was a first single user instance for testing if I really change from mastodon to friendica. It worked, but it wasn't performant.
I think that my problems at the new server came from problems with the database that began at the shared webspace.
I dumped and restored the database there for testing the recovery scenario and the virtual view-tables couldn't be restored. I installed them by using the friendica cli and then used that database furthermore. And I dumped that database again to migrate to the new vps.
Beside the problems from friendica being under attack at the new server, I think the database once not restored properly was the reason I had to reinstall friendica from scratch 🤷♂️
For Firefish I was able to restore from an earlier backup than the most recent and it was able to recreate most of its state and carry on working. I don't know if Friendica can do that. A caveat being I don't have anyone else with a local account and could live with my profile pic reverting etc.
This is the resource usage for Firefish if you want to compare (Meilisearch is not necessary, it's for the full text search and I should probably switch to the much lighter weight Sonic):
Ideally you'd just go back a day or a week or something. I had a corrupted backup archive on my cloud storage so I restored from an older backup kept on my local machine that I sync less often and it didn't revert everything.
If you do end up starting over I wonder whether it's worth looking for instances of the problem happening with postgres in case the issue is mysql/mariadb specific for some reason.
There are two separate problems described there. But If you are not running an old version of MariaDB (<10.6) you might have the same problem I experienced. I solved it by increasing the innodb_buffer_pool_size to 16 GB, original description here:
This is something so weird and I still do not understand it. Context. This is our instance https://social.trom.tf . Our database is around 30GB in size. We use a daemon for Friendica with these con...
The answer to this question also interests me. In my case it happened at the 2023.05 update. This fixed my issue on a system with 8GB RAM. Did it help in your case? @Ian Molton
I can say that increasing it from 2G to 4G (still less than the 8G default in debian) made a substantial difference.
I *cannot believe* that a minimum requirement for a database of this type is over 8GB.
That's insane. I wrote production systems using mysql in 1998, back when 8GiB RAM was a total fantasy, that ran large scale SMS <-> Email gateways, and my machine could saturate 100MBit ethernet.
it had a 266MHz PII and 128MiB RAM, running FreeBSD IIRC.
But is innodb_buffer_pool_size really in RAM? Because my server only has 8GB of physical RAM and it still works "good enough" for my single user instance - while having the innodb_buffer_pool_size set to 16GB.
I restarted mysql when it stopped working (again), and after it restarted, loading the home page used well over 1GiB of RAM (I watched the RSS in top as the queries executed).
At this level of resource consumption, my instance can scale to maybe 2 simultaneous users.
This is clearly insane.
Can anyone actually help, or is this just going to be another case of the problem going unfixed?
If so, has anyone got experience migrating their timeline from friendica to something that doesn't require a datacentre to support more than one concurrent user?
I'm going to use a small SBC running debian as the router, possibly an orangepi.
I'm trying to find a nice managed switch and some APs and cameras. I don't want to run a ton of power leads around, so POE support is a must. 16 ports would do, more is nice. A couple of SFP ports would be nice too.
I'd prefer to run OSS firmware on everything, and Im willing to spend *some* money on getting that.
So I need
* a managed switch with VLAN capability, POE, 16-24 ports * 2-3x APs that can handle VLANs and run on POE. WiFi6 would be nice but 5 will do. * Cameras - perhaps one PTZ, and couple of fixed position.
If a router with open firmware *plus* all of the switch requirements exists, that would be an option too (esp. if it can run a proper Linux, eg. Debian). Incoming connection is PPPoE on Gigabit. Such a router should be capable of handling up to a gigabit on WAN (for when that comes down in price).
I am running VyOS (the community version) for routing and like it generally. I have no idea if an ARM version of it exists though.
I would probably recommend Ubiquiti for APs but they are not open source. Likewise perhaps Mikrotik or Ubiquiti for switching (for that, I run used Arista & Cisco enterprise atuff), but none of that is open source either (and the enterprise stuff is loud and power hungry). I am not aware if any good open source switching but I could be unaware of stuff.
I have to admit I was *shocked* when searching for open source switches came up with only one (defunct) project, and someone who stuffed an arduino into a switch so they could reprogram the switch chips regsters on the fly, granting themselves management on an otherwise unmanaged switch...
More than 1,000 children have reportedly been killed in 11 days of airstrikes on Gaza – one child every 15 minutes – with children making up a third of total fatalities in Gaza.
the sad thing is long ago I once said Gaza was a rather authentic recreation of the Warsaw Ghetto. Now it seems we will be witnessing an authentic recreation of the raising of the Warsaw Ghetto, too.
They want to expunge Palestine and Palestinians from existence. They always have. They haven't even made much effort to hide the fact, either. And yet the West continues to pretend it's never been happening.
It's wrong to kill children wholesale as is happening in Gaza, or retail--3 here, 15 there, 22 in another place, day after day--as we kill them in the USA. In 2021 we averaged 13 kid gun deaths per day in this country. * PER DAY. PER FUCKING DAY. We shrug; we let it continue. It seems there is no enough.
The State of America's Children®
2023 Gun Violence
Gun violence remains the number one cause of death for children ages 1-19 at a rate of almost 5 in every 100,000.
Please be careful. By blaming 'the Jewishness' of Israel's government, you are playing right into the hands of those who suggest all on the Left are anti-Semites.
@wootube Who the hell is blaming “the Jewishness of Israel’s government?” Where the hell did you read that because it sure as fuck wasn’t in anything I wrote. Fuck right off with your insinuations of anti-semitism because I have the audacity apparently to decry the slaughter of civilians.
Only people accusing you & others are fucking zionists bent on power & who don't care how many ppl die.
I will say - those folks are evil & they might as well sleep with terrorists because they ignored terror upon Palestinians for DECADES.
Not all of us bought into Israel propaganda shit. It is really obvious now; who bought it & who took the free trips on Israel(USA funded) trips though.
I respect my Jewish friends who refused the free trips & tried to challenge Israel govt.
@wootube I've been reading your posts and I agree that you haven't said anything antisemitic, I think your ire should be directed towards Hamas. Israel does everything it can to minimize civilian deaths and Hamas does everything it can to maximize civilian deaths.
@Inekeduzz Yes, it’s horrible. And Hamas are barbarians. No one (except some unconscionable tankies) is arguing otherwise. This does not justify an entity that purports to be a democracy acting in the same way or worse. It does not justify the mass slaughter of civilians with genocidal intent.
, "Between February 24 and March 21, 2022, sixty-four medical facilities and their personnel were targeted by Russian forces in Ukraine, the World Health Organization (WHO) reported. The facilities were being hit at rate of two to three a day, inflicting 15 deaths and 37 injuries. By April 8, 2022, WHO had confirmed 91 attacks."
Where is the outrage at Ruzzias ethnic cleansing of Ukrainians? Note that this WHO report is only for the first month of Ruzzias invasion.
Thanks for sharing this. Bloodlust is a good word for this situation and I find that terrifying. People want revenge, not solutions, and the Israeli government is going to use that desire as justification to kill as many people as they can.
500 dead in a hospital and people still try to argue that this is just about “fighting Hamas.”
“La matanza en el Hospital Al-Ahli no tiene precedentes en nuestra historia”, ha declarado el portavoz de la Defensa Civil Palestina, Mahmoud Basal. El presidente de la OMS ha condenado el bombardeo.
An Israeli air strike killed at least 300 people at a Gaza City hospital on Tuesday, authorities in the Palestinian enclave said, and the United Nations said an Israeli strike also hit one of its schools being used as a shelter.
as much as I honestly think that Netanyahu should face a process for being a war criminal for a thousand reasons (people in Gaza are drinking salty water because the access to fresh water was cut), *this* particular case is not confirmed (yet) to be Israelian responsibility. And as the situation is already a mess we should be careful not trying to spread disinformation.
@CiaobyDany The Irish Times is the newspaper of record in Ireland so it’s not like I was quoting some random social media account. They’ve now updated the article to say “strike” as Israel is saying it wasn’t them. Regardless of who it turns out to be, it’s a bloody tragedy.
absolutely and I have specified it in the toot as well. I also had thought it was Israel as well at first. I think that anybody should do what they can to not spread misinformation. Antisemitism and antiislamism are already rampaging enough as it is.
IDF were saying they needed hospitals vacated so they could bomb the Hamas hiding there. Now they bombed a hospital and are saying Hamas did it to themselves. They don’t even bother to keep their story straight cause they know there will be no consequences.
Israel's "final solution" has been in progress for decades. They knew the Hamas attack would eventually happen, but in their apartheid arrogance they just did not expect that much loss. No matter, their failure gives even more motivation for the always intended extermination of Palestinians from the land. The acceleration of the mass murder plan has every so called democracy's leadership as complicit. And of course, there will be no holding any of these people and governments accountable.
Has a hatred of jews always been present in our societies? From anti-Judaism to modern anti-Semitism, this documentary series explores the multiple facets of the phenomenon, from its origins to the present day.
The Irish Times has updated its headline to “Hundreds dead after Gaza hospital strike as conflict continues to escalate” as Israel denies it was responsible for that particular strike, blaming a failed rocket launch from Gaza.
This is madness. And I fear it’s only going to get worse. And my heart breaks at the senseless loss of life and human suffering.
Also see: Hananya Naftali, media advisor to Netanyahu, first posted it was an Israeli strike on X, then deleted his post and posted about a “mysterious explosion” an hour later.
did you personally see the tweet screenshotted on the left side of this image? or how do you know it's authentic? just asking since twitter/x screenshots are easily faked.
@scott See link. He admits to it in it. (Screenshot is from Reddit thread but other versions also exist online. The authenticity of the original post, afaik, is uncontested, even by the author.)
The Irish Times has updated its headline to “Hundreds dead after Gaza hospital strike as conflict continues to escalate” as Israel denies it was responsible for that particular strike, blaming a failed rocket launch from Gaza.
This is madness. And I fear it’s only going to get worse. And my heart breaks at the senseless loss of life and human suffering.
TBF, the US infamously started "The War on Terror" and invaded a couple of countries in the process, so I don't see why they'd be against what's happening in Gaza. The west watched, did nothing, if not aided, back then, and it's watching and doing nothing now.
Unfortunately, the majority of people are like that. Arabs don't care about what's happening in Ukraine either. The west can get on the highest horse they find, but the truth is the majority of people just suck, western or not.
The Irish Times has updated its headline to “Hundreds dead after Gaza hospital strike as conflict continues to escalate” as Israel denies it was responsible for that particular strike, blaming a failed rocket launch from Gaza.
This is madness. And I fear it’s only going to get worse. And my heart breaks at the senseless loss of life and human suffering.
I am from the west and I am not fucking cheering. It’s the same fucking situation as Russia. The oligarchy own everyone here. They profit from the killing they always want more.
I know you’re getting traction on this, but it should be edited. Israel denies EVER targeting a hospital and states Islamic Jihad was firing missiles into Israel when the hospital was hit. Those missiles reportedly have a 30% failure rate. 30%!
Claiming Israel is committing genocide is irresponsible, antisemitic, and leads to death. Be responsible
Claiming Israel is committing genocide when Israel is committing genocide and calling Palestinians “human animals” while murdering 1,000 children is not antisemitic. It is called being human and being appalled by the barbaric slaughter of human beings regardless of who is carrying out the slaughter. And you will not shame me into ignoring the humanity of the Palestinian people.
If you generally like manga/anime, I would say it's a classic. But I also don't know if it's for everyone (coming from outside the manda/anime world) ahah. I think you can try to watch a few episodes and see if you like it. I personally loved it :)
The Labour party loves to talk about "reforming" the NHS – but does it really mean something else? Rivkah Brown heads to party conference in Liverpool to inv...
Cory Doctorow has some interesting takes on why using a copy left license and the Fediverse at the end of his DEF Con 31 talk. But you don't have to scroll to the end, the rest of the talk is nice to listen to as well 😉
The enshittification of the internet follows a predictable trajectory: first, platforms are good to their users; then they abuse their users to make things b...
""" Hamas militants entered Israeli territory in the early hours of this morning, appearing to take control of multiple communities in southern Israel. Fighting is still ongoing in areas of southern Israel.
“We are at war and we will win it,” Israeli prime minister Benjamin Netanyahu said in a video address to the country. """
I’m seeing a lot of people share conspiracies about how the government is going to activate some vaccine virus using 5G technology today. If you are posting this nonsense, please stop and seek help. You are deep in some weird, dark place, and you don’t even know it. (Also, it’s not the U.S. government, it’s the Romulans, and it’s not 5G, it’s transporter technology. Please get your conspiracies straight.)
Conspiracy theories have to be the most novel form of pathological denial. People will believe anything if it means that they don’t need to fear a real threat.
They are? Hmmm, I thought it was Hugh Darrow broadcasting a rogue signal to all vaccinated people, from his fancy office at Panchaea ("Deus Ex: Human Revolution") 😂
I thought the nanoparticles were supposed to give me my own wifi and 5G (or was that last year's rumor). I'm very disappointed to report that I do not have my own personal internet emanating from my body. Very disappointed.
these are not people, these are troll networks run by small groups pushing out large volumes of #socialmedia posts on every trend. This is most of the amplified content on social as platform #algorithms like that crap. In this case the Gov emergency alert system test was a perfect opportunity to spread #conspiracy. #5g
For 15 years, Brent Lee spent hours each day consuming ‘truther’ content online. Then he logged off. Can he convince his former friends to question their worldview?
are you implying that there is time travel technology embedded into the vaccines, and once activated, they will all send us back in time to the dinosaurs and we all get eaten? It sounds like that's what you are saying. I'm going to run with this one.
no, I heard it was a rogue Ferengi trader using tachyon beams to activate cell level micro bots in order to create submissive positronic brains and bigger boobs in human females.
No joke, this entire conspiracy theory is literally the plot to Star Trek Picard Season 3:
The Changelings infiltrate The Federation to modify the transporters to inject modified genetic code (The Vaccine), and then once it's activated by a signal from the Borg Queen (5G), all Federation personnel turn into subservient zombies.
Honestly, I'm a little shocked at how lazy the conspiracy theorists are becoming. 😂
The Romulans will attack from beneath the disk where we cannot see them. Their boring machines could pop up anywhere (other than Australia which as we all know is imaginary)
Those conspiracy theories about the National Alert System, concocted by anti-government crackpots, make no sense at all. Conspiracy theories are all a figment of their overactive imagination. Those cranks might want to do something constructive and make a killing writing fiction.
Was this because of the emergency alert test today? It was announced in advance, so it had to be a plot, I guess. Set everyone's phones off in a meeting I was in.
All any government needs to do in order to control the minds of its people is make something akin to fox "news" and just spew propaganda at them for a long enough time that the people lose the ability to parcel the truth from the lies. I didn't want 1984 to be true but here we are.
It's simply right wingers everywhere pretending to embrace post-modern ideas of the relative truth of each person while spouting crap they know is complete bollocks and defending the free speech rights of their donors and colleagues to spout even stinkier shit.
Great way to distract from the wealthy fleecing the poor.
I'm listening to The Ragged Trousered Philanthropists while typing this. My deja vu is at eleventy.
Emily challenges the Conservative MP over speeches made at the Tory Party Conference. These include the Transport Minister, among others, who repeated a fals...
we have a hell on earth. this side of the cd is working .. this part of the song hits hard. this page belongs to my former diary. this person is not existing. this cell phones are so fake.
I assume these 5g conspiracy nuts are unaware that the 5G frequency varies a lot between countries.
In the UK they occupy frequencies that were used by broadcast TV. They weren't dangerous for the many decades that they were broadcast all over the country. Probably the same in many other countries.
This material is so hilarious that it should be in a comic strip. (Not the comments made by anti-vaxxers, those are deppresing. Research and the court of justice are similar. Both sustain themselves on evidence -physical & statistical- not opinion).
I placed my phone closer to my head, in the vain hope that it would increase the potency of my zombification. Alas, it appears to have had completely the opposite effect.
5G what? Stupid is what Stupid does. We are vaccinated and want good cellphone service with high-quality data. Sometimes, it's hard to understand why things work, but the scientific explanation is all we have.
#OnThisDay, 4 Oct 1936, women join the Battle of Cable Street, in London's East End, protesting against a fascist march through the area. Of the 79 people arrested, eight are women, including Blanche Edwards (pictured).
How is this man and his social network, car company, space rocket company, and satellite internet company still considered socially acceptable? How is it that these enterprises aren’t being nationalised as we speak? How have we built a system that franchises a handful of billionaire psychopaths at the expense of everyone else?
@aral Taking you question literally, we have bought into a weird conception of property, such that property rights are fundamental. That distorts everything.
We need to get back to thinking that property ownership is just a permission to use something in certain ways. Limited and dependent.
@aral FWIW one of the best books for showing how weird it is is *The Book of Trespass* by Nick Hayes. Recommended for everyone's awakening and a great read.
@aral I don’t think nationalizing them is the right way to go but we definitely need to break up monopolies. The question is, how? The laws are not working as intended and with our SC it seems hopeless.
Updated October 2021 One of the best parts of Halloween is decorating with pumpkins. After the trick-or-treaters clear away and Halloween is officially over, don’t toss your pumpkins in the … Read more
Content warning: PSA for health tracking apps of any kind:
If you have any influence for any fitness app or accessories (watches etc), you don't only need an Athlete Mode , you also need some form of disability mode.
We want to be able to track our health, without being asked to Go further, all the time, without the "whohooo you did 10k steps today, keep going" when we shouldn't take more than 2k/ day. Etc etc (feel free to leave examples if you're reading this)
Let us turn off any kind of gamification, turn off anything asking us to go further, any ui that's telling us after 1 week that we're gaining or losing weight.
Whoever is able to provide this first? Will get a huge new market share because we'll fucking recommend the shit out of your app!
Talking to a friend who’s in the hospital and had a sign on their door requesting masking, for their safety. Most staff only wore paper masks, if that. (Doctors were worst at masking at all.)
Yesterday friend had a bit of a sore throat, and was immediately COVID tested and now the door sign is “heightened COVID risk” and everyone is wearing N95s on entering.
They’ve decided to have a sore throat until discharge, since apparently the only way to be protected is to pose a risk to the staff.
@jentrification My history has made me very sensitive for *any* sign of parents using their children. Whether that is using them as props in an instagram-mommy career or as mouthpieces in an activism movement or expecting them to make a real contribution to household tasks or using them as friends when you are lonely.
It is not your children's job to solve _your_ problems. And when your kids say 'it is okay, we WANT to help' that is not real, but them trying to make you feel good.
I feel this, too. I was "complemented" as a kid for being so grown up and communicating so well with adults.
Why was that? Was it because I just had good communication skills? Or was it because I didn't have any friends and adults were the only people who would talk to me?
There are many reasons why I grieve for the little girl who never got to be. Only some of that is because everyone thought she was a boy. But also, she was forced to be a "man" at age 12 or 14 because her dad wanted clones of himself, not healthy, independent children.
I really didn't expect this would resonate with so many people (and also get me harassment for some reason? Some people are just that fucked up I guess.)
I've heard several stories today from people who have a similar experience, some for similar reasons other for entirely different ones. And I'm glad I could give some of them the words to describe what they feel. That's really the best outcome of writing about these things for me and why I do it, because I know I would have needed someone to tell me the very same things like 20+ years ago.
I got that shit too. Got parentified from early age, trying to do a balancing act of looking after my younger siblings and trying not to land in hot water with my parents when I couldn't contain what had happened.
and all the older ladies just circle-jerked about how 'mature you are for your age'. Well the fuck you expected, when a 10 year old is literally on hook for younger siblings' food, clothing and education, plus keeping them out of parents' way.
that did a number to my personality, some call me childish these days but it was like the world had been lifted off my shoulders when I got out of that. it took a long time for me to experience video games and basically a lot of other 'kid stuff'.
Nobody would ever want to live beside a data center. They are big ugly warehouses. They are very noisy. There are air pollution dangers. They take huge quantities of water and electricity. They bring very few jobs. “Just a dozen people, or two dozen people, run a facility that is consuming as much electricity as a small city,” Steven Gonzalez says. “A data center life is between five and twenty years. This is not a permanent industry. It is extractive, like mines.” https://gerrymcgovern.com/communities-against-the-injustice-of-data-centers/
Between five and twenty years? Is that correct, and if so, why is that? I know they go through hardware like crazy, but why would they move sites so often?
@mfru I see. The frame of the building and walls are fine for decades, but due to technological progress the wiring, cooling systems and other hardware needs to be replaced. It is cheaper to tear everything down and rebuild anew with the recent efficient technology than to retrofit it in an old building.
These tiny creatures are losing their battle to survive.
The Rufous hummingbird lost two-thirds of its population since 1970, according to the 2022 State of the Birds report.
These tiny creatures are one of 70 bird species on the “Tipping Point” list that will lose another fifty percent of their populations in the same time frame if conservation doesn’t improve.
I'm asking people to go and review Cllr Mandy Clare (CWAC) IMDb page.
Someone has already updated the links to this toxic and dangerous public figure's "home page" on there and on Wikipedia, but it would be nice if we can cancel out the hundreds of positive reviews of the hate documentary "Adult Human Female" on IMDB ( https://www.imdb.com/title/tt23844954/?ref_=fn_al_tt_1 )
I'm asking people to go and review Cllr Mandy Clare (CWAC) IMDb page.
Someone has already updated the links to this toxic and dangerous public figure's "home page" on there and on Wikipedia, but it would be nice if we can cancel out the hundreds of positive reviews of the hate documentary "Adult Human Female" on IMDB ( https://www.imdb.com/title/tt23844954/?ref_=fn_al_tt_1 )
In March of 2023 a family-friendly drag show in Wadsworth OH was threatened by anti-LGBT protesters including Proud Boys, Patriot Front, and Neo-Nazis. Show organizers asked for volunteers to help protect the show and its attendees. Leif was one of many people from Central Ohio who answered this call for volunteers. On the day of the show, attendees and supporters were outnumbered nearly 2-1 by anti-LBGT demonstrators from all over Ohio. While escorting families out of the event, Leif was assaulted by anti-LGBT protesters who hit him in the head with a megaphone. Police stood by and allowed this to happen. Leif feared for his life and the lives of the people he escorted - including children and a person with a service animal - and he took swift action to end the attack. FOR THIS JUSTIFIED ACT OF DEFENSE HE IS BEING CHARGED WITH THREE COUNTS OF ASSAULT!
Do trans people have the same self-defense rights as everyone else? The law throughout the whole US allows a person to use force in defense of human life. Ohio is a ‘stand your ground’ state. This means a person has no “duty to retreat” and may use force in self-defense or defense of others, if they feel their life is in danger. The police and the state of Ohio have chosen to prosecute a trans man with criminal charges instead of the transphobes who assaulted him. Weare going to fight back using any and all legal means necessary.
We need YOU to help us fight to the finish! Here’s what you can do:
Donate at http://spot.fund/lwhhjsc Good legal representation costs good money! Every donation, big or small, will help us take this case to trial and fight to the finish.
Share these graphics and the Spotfund link to help boost our reach!
Pack the court on October 11, 1:15pm, at Wadsworth Municipal Court!
Jessica Burgess, a #Nebraska mother accused of helping her teenage daughter use pills to end her pregnancy, was sentenced on Friday to two years in prison.
![%%OPL: ‘w}wo‘ think Hhe | @I VQCdne}. Will‘ (=Sl / f AR { oppor‘l‘um@_ \](https://social.weyr.org.uk/photo/preview/1024/599517 "%%OPL: ‘w}wo‘ think Hhe | @I VQCdne}. Will‘ (=Sl / f AR { oppor‘l‘um@_ \")
Friendica Support reshared this.
Ian Molton
•Crazy-to-Bike likes this.
Crazy-to-Bike reshared this.
Friendica Support reshared this.
Sean Riley
•Make sure you block those activity pub trolls, they will grind you to a halt.
Also if this is a new instance give it time to settle down. Friendica does alot of work in the first week or two to build out its understanding of other hosts, their accounts etc.
Increasing the workers is also helpful. And run it as a daemon.
I found a that shutting down mariadb each evening for backups and then restarting helped as well.
Crazy-to-Bike likes this.
Friendica Support reshared this.
Ian Molton
•Friendica Support reshared this.
Ian Molton
•Crazy-to-Bike likes this.
Friendica Support reshared this.
Sean Riley
•Friendica Support reshared this.
Ian Molton
•Yeah, I blocked that one ages ago.
Is it really the only one? Seems unlikely.
Friendica Support reshared this.
Crazy-to-Bike
•@Sean Riley @Ian Molton
I have these domains in my server block list since my instance was under attack from *.activitypub-troll.cf:
*.activitypub-proxy.cf
*.activitypub-troll.cf
*.gab.best
*.misskey-forkbomb.cf
Friendica Support reshared this.
Sean Riley
•That looks pretty good, how many jobs are in the worker queue? How is CPU utilization? Memory usage?
Friendica Support reshared this.
Ian Molton
•Friendica Support reshared this.
Sean Riley
•Your list of 4 troll exclusions
Friendica Support reshared this.
Ian Molton
•I already had the .cf one
none of them were the issue in this instance though.
Friendica Support reshared this.
jesuiSatire …ᘛ⁐̤ᕐᐷ
•> Your list of 4 troll exclusions
* me
* myself
* my first secondary account
* my second secondary account
@dogriley
@spyro @crazy2bike
Friendica Support reshared this.
Michael Vogel
•Friendica Support reshared this.
Ian Molton
•I think the workers were pushing the machine into swap when large qeries are caused by the frontend (I've now seen a query over 600MiB)
There is no way the system can handle >1 user if its going to cause 600MiB of disk reads every time they access the site!
Either something is wrong, or there is an incredibly inefficient query running somewhere, or a query is being run over an unexpectedly huge data set - but I have no way to know.
Whatever it is, it's crippling the machine - it seems to cripple it gradally enough that it doesnt immediately OOM (sounds long-query-ish to me), and it doesnt prevent more workers running - the machine is being killed by multiple threads seemingly eating ram whilst being "blocked" by a large running query.
Eventually the machine gets so bogged down that it can't serve anything. I can still log in (just - last time it had a load average of >30), and if I kill / restart mariadb, it will carry on, but it seems to be living right on the wire - a good push will cause it to start losing the battle and bog down again.
Crazy-to-Bike likes this.
Crazy-to-Bike reshared this.
Friendica Support reshared this.
Michael Vogel
•I like to go for an analytic approach. First please let us trace the requests. There we can quickly distinguish if the delays are possibly caused by a lack of available /Apache/Nginx/PHP connections or if the database is the bottleneck.
Depending on the result, we then can change some parameters.
Ian Molton
Friendica Support reshared this.
Crazy-to-Bike
•@Ian Molton
I had this problem too. But even banning those spam domains and cleaning the database from their entries didn't have the effect that my server worked fine again.
The load stayed high and the responsing time was bad.
So I'm sure that in my case the change to another server with dumping and restoring the datebase was part of the problem.
Friendica Support reshared this.
Michael Vogel
•Please have got a look at the requests, that your system receives. Switch the log level to "debug" and then run that command:
tail -f friendica.log | grep -a Request.*'"runFrontend"'(Of course you have to change it to the name of your log file)With that command you will see incoming requests. "Request received" tells you, that the request just arrived, and both "Request processed sucessfully" and "Request processed with exception" tells you, that the processing is done. BTW: Don't be confused by that "with exception", if the response code is in the 200 range, then everything is fine.
With these log entries you will see, if there are delays before the processing starts or during the processing. Depending on that, we should see, what to do then.
Ian Molton likes this.
Friendica Support reshared this.
Ian Molton
•There is a LOT of activity when I do that - I can't really tell whats going on.
I can see that most things seem to be getting 200 responses, but there can be long pauses.
Friendica Support reshared this.
Michael Vogel
•Friendica Support reshared this.
Ian Molton
•How?
Is there a set of tools? is there a debugging guide?
Friendica Support reshared this.
Michael Vogel
•Friendica Support reshared this.
Ian Molton
•Michael... please remember...
I ***dont know what I'm looking at*** here.
I don't know how friendica works internally.
things like... "What is a Gserver? Why is it updating? Whats the difference between updating gserver*s* or in the singular?
What is Delivery? Why is there APDelivery?
What do all the things in the queue *MEAN*?
I can add a "grep network" to the suggestion you made earlier, but it //tells me nothing//.
Currently, My instance is responsive enough to the "home" button, bt the network page is still causing 504 errors.
This *reeks* of runaway query, but I dont have any idea where to start looking for it.
Friendica Support reshared this.
Michael Vogel
•tail -f friendica.log | grep -a Request.*"GET /network".*'"runFrontend"'This will show you all the network requests. You will see the first entry, once the system received the request. When you don't receive an entry there (or only sometimes), then please consult your apache/nginx configuration and increase the number of possible connections or possible php processes.
Friendica Support reshared this.
Michael Vogel
•tail -f friendica.log | grep -a 'Request processed sucessfully'.*'"runFrontend"'There you can see the successful frontend requests and their performance (in the field "duration"). Perform some frontend requests on the network page and see if the duration is high (multiple seconds) or not.
Friendica Support reshared this.
Ian Molton
•Trying this, I see nothing a the start of the request, and ~20 *seconds* later, I see the following.
I can't see any "duration" field.
2023-11-12T15:12:15Z app [DEBUG]: Request processed sucessfully {"response":200,"address":"81.187.24.115","request":"GET /network HTTP/2.0","referer":"https://social.weyr.org.uk/admin/queue","user-agent":"Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/113.0"} - {"file":"App.php","line":718,"function":"runFrontend","request-id":"a6f17cc40134fbcb0a68ccd553c15016","uid":"750f76","process_id":1224165}
Friendica Support reshared this.
Ian Molton
•Friendica Support reshared this.
Crazy-to-Bike
•@Ian Molton
I had such problems after migating to another server with dumping and restoring the datebase.
Server load up to 14 with 8 cores, long loadinig times of the pages, no reload of the timeline with apps like Fedilab or Tusky.
At the end, I made a fresh installation at the new server and now everything is smooth.
Server load of < 1, quick responses, renew of the timeline in apps.
Friendica Support reshared this.
Ian Molton
•Yeah, but I've got like 30 users - periodically deleting the system and reinstalling it is not an option.
Frankly, if there is an O(N^2) or worse algorithm in play, then this is only going to happen again.
The culprit is looking like low memory - not something I thought would be a problem on a machine with 4GiB of RAM, with little else to do...
Friendica Support reshared this.
Ian Molton
•Update:
I've not done much - I've increased the php-fpm timeout in nginx so that some of the longer requests complete, bt this is looking like a database issue.
The VPS i'm running friendica on is essentially idle otherwise, has 4GiB RAM and two decent CPU cores.
Requests into the server aren't taking up much of its time, bmon shows ~4-15PPS a burst or two of 50-300PPS every 60 seconds or so. Peak bandwidth is about 2MB/s typical is more like 10-100K/s
Last night, mariadb crashed - I had increased the size of innodb_buffer_pool_size from 2 to 16G (yes, I know this is > RAM+SWAP). I reduced it down to 4 again, and it's seemingly ploughed through the worker queue as of this morning, but the whole site is still VERY slow.
Right now, from an adminn point of view, I see the biggest problem in friendica is that there is literally ZERO ability to introspect its behaviour.
I can look at the "worker queue", but there is no way to know what any of that means short of dismantling the code, which I dont have time to do.
I can't search the worker queue for issues in any meaningful way, and even if I could there is no mechanism to inspect / suspend / pause / cancel entries in the worker queue.
If I could suspend the entire queue, and selectively "gate through" certain entries, it'd be enormously helpful.
Lastly - of the "known" threats out there like activitypub-troll, is there a list of workarounds or a blocklist? HOW can we identify these malicious sites and keep our instances safe?
Friendica Support reshared this.
Ian Molton
•Friendica Support reshared this.
Crazy-to-Bike
•@Ian Molton
As a reference point:
For testing I installed friendica on a shared webspace with 1 gb ram and 16 customers per core as single user instance.
It worked, but it was not perfect for a permanently productive instance.
Now my instance (until now no other users) is on an vps with 8 cores and 12 gb ram with a load < 1.
Friendica Support reshared this.
Ian Molton
•was that including the database? 1GB RAM? how many users?
I couldnt get friendica to stay online in that scenario - the database (mariadb) just ate all the RAM.
Friendica Support reshared this.
Crazy-to-Bike
•@Ian Molton
Yes. As described it was a first single user instance for testing if I really change from mastodon to friendica. It worked, but it wasn't performant.
I think that my problems at the new server came from problems with the database that began at the shared webspace.
I dumped and restored the database there for testing the recovery scenario and the virtual view-tables couldn't be restored. I installed them by using the friendica cli and then used that database furthermore. And I dumped that database again to migrate to the new vps.
Beside the problems from friendica being under attack at the new server, I think the database once not restored properly was the reason I had to reinstall friendica from scratch 🤷♂️
Friendica Support reshared this.
Sean Riley
•I wouldn’t run it on a shared server. While possible I found resource requirements high to have a performant instance.
Friendica Support reshared this.
Ian Molton
•Sean Riley likes this.
Friendica Support reshared this.
alastair87
•Friendica Support reshared this.
alastair87
•Friendica Support reshared this.
alastair87
•Friendica Support reshared this.
Ian Molton
•Friendica Support reshared this.
alastair87
•Or it didn't load at all when I tried it.
This is the resource usage for Firefish if you want to compare (Meilisearch is not necessary, it's for the full text search and I should probably switch to the much lighter weight Sonic):
Friendica Support reshared this.
alastair87
•Friendica Support reshared this.
Ian Molton
•Friendica Support reshared this.
alastair87
•Friendica Support reshared this.
alastair87
•Friendica Support reshared this.
alastair87
•Friendica Support reshared this.
alastair87
•Friendica Support reshared this.
Ian Molton
•Friendica Support reshared this.
AndiS 🌞🍷🇪🇺
•Have you seen the following issue on Github?
https://github.com/friendica/friendica/issues/13373
There are two separate problems described there. But If you are not running an old version of MariaDB (<10.6) you might have the same problem I experienced. I solved it by increasing the innodb_buffer_pool_size to 16 GB, original description here:
https://github.com/friendica/friendica/issues/13373#issuecomment-1712594367
@Ian Molton
Very slow after the latest 2023.05 update · Issue #13373 · friendica/friendica
GitHubFriendica Support reshared this.
Ian Molton
•Hi, this was the reason I tried increasing innodb_buffer_pool_size to 16GB...
Whats the cause of this (colossal!!!) increase in memory usage / requirements?
Friendica Support reshared this.
AndiS 🌞🍷🇪🇺
•Ian Molton likes this.
Friendica Support reshared this.
Ian Molton
•I'm not lucky enough to have any way to try.
I can say that increasing it from 2G to 4G (still less than the 8G default in debian) made a substantial difference.
I *cannot believe* that a minimum requirement for a database of this type is over 8GB.
That's insane. I wrote production systems using mysql in 1998, back when 8GiB RAM was a total fantasy, that ran large scale SMS <-> Email gateways, and my machine could saturate 100MBit ethernet.
it had a 266MHz PII and 128MiB RAM, running FreeBSD IIRC.
What the actual?
Friendica Support reshared this.
AndiS 🌞🍷🇪🇺
•I'm with you on "what the actual...."
But is innodb_buffer_pool_size really in RAM? Because my server only has 8GB of physical RAM and it still works "good enough" for my single user instance - while having the innodb_buffer_pool_size set to 16GB.
@Ian Molton
Friendica Support reshared this.
Ian Molton
•I don't know. But 2GB is not enough, and 8+GB seem to be too much...
The actual database is far larger anyway (41G) but the machine is 64 bit, so mapping that in (virtual) memory is a non-issue.
Friendica Support reshared this.
Ian Molton
•Friendica Support reshared this.
Ian Molton
•Just tested after midnight for the first time - 680MiB of mysql reads caused by opening /profile/spyro
That seems like ... a lot?
Ian Molton
I'm still experiencing extreme slow page loading.
I restarted mysql when it stopped working (again), and after it restarted, loading the home page used well over 1GiB of RAM (I watched the RSS in top as the queries executed).
At this level of resource consumption, my instance can scale to maybe 2 simultaneous users.
This is clearly insane.
Can anyone actually help, or is this just going to be another case of the problem going unfixed?
If so, has anyone got experience migrating their timeline from friendica to something that doesn't require a datacentre to support more than one concurrent user?