Database Down!


Unfortunately, it appears the database is currently inaccessible.
This may be for maintenance, or the server is just starting, or some other error. Please try again shortly. If this error persists, please contact an administrator in the #earthempires IRC channel.

Been getting this error quite frequent in the past few days.

same

I attempted an emergency fix. Are you still seeing the problem?

Haven't today

I got it while running turns in coop right now as I post this. it was also very slow while using turns in tourney but no error.

thx for trying to fix it though

It seemed immediately better but it did have more issues through 5:00-6:00 game time. It's certainly improved, but it does seem like there's still an underlying issue.

personally haven't seen it happen again in the past 12 hours, hope it's fixed. thanks

The issue isn't fully fixed, but game performance is significantly better since the change 12 hours ago.

It just got a whole lot worse again :/ Unplayable in Alliance for me. Really slow and error after error

Still getting the same error inconsistently.

it happened again just now for me, didn't experience it yesterday. also happened in the forums

Yeah, I can confirm it still happens; however, it appears to be less frequent based on observation, and not on actual data points to support that.

Error is back :(


Database Down!



Unfortunately, it appears the database is currently inaccessible.

This may be for maintenance, or the server is just starting, or some other error. Please try again shortly. If this error persists, please contact an administrator in the #earthempires IRC channel.

We are unlikely to see any further relief until Friday at the earliest. I apologize for the inconvenience and ask for your patience.

Any advice on how to best go around with it in the meantime? It seems to come at certain times. If I try again an hour later, sometimes the issue is gone? Is it related to bandwidth of players, bots or just traffic at the host side?

In my experience if you wait a few minutes then it's usually fine. The issue is related to I/O performance on the server.

Weird to see database go down randomly.

Want to elaborate on the issue? Be as technical as you want. :)

I assume it's not actually down, but that there is a request timeout and so the website displays that error. There have been disk failures in the past that looked very similar to this, with varying solutions (all of this per qz in discussions over the last decade).

bonus

wouldnt it be cheaper to run this stuff on aws at this point?

Probably not. Cost savings from AWS are usually from using their apps, and scalability.... not by just throwing a VM in the cloud.

Possible to fix the error before set ends in Alliance?
We still have some netters in there and they need to destock smoothly without any error and slowness.

Just out of curiosity I wonder how much it would cost for a site this size w its traffic in the cloud. I bet you’d still wanna run bots locally to save compute.

still happening to me as well

Seems like someone decided to delete my post.

Hopefully there's a solution soon in time for end set.

Yeah no fluff I deleted it, don't post private information about our servers in public for no reason.

I seen Rasp's post and it wasn't anything private TBH. Many people have speculated about this. We've seen these issues before and Qz was transparent about it in the past.

You shouldn't be so seductive Slag, Rasp is a respected member of this community. Control your emotions.

I have not had any issues for 2 days now. Does occasionally work really slow in FFA, but no Database down errors in the past 2 days

Rasp is an admin for a clan hosting site that does way too many feed hits (over 200k per day). That puts him in a position to help with the recent performance issues that we've had. Unfortunately, he is leaking private information in this thread instead of fixing the issue with his clan site.

I am obviously not going to respect that behavior. In fact, I find it to be very irritating. I'm sure that you can understand why.

The need to hit the feed often is a design flaw with the feeds themselves IMO. I would prefer to see a websocket feed vs sending HTTP polling.

My gut feeling would be that it would save CPU by not having to establish the full HTTP pipeline on multiple back-to-back requests all the time X however many sites there are running.

The news feed for example has a trottling limit of 1 second. To get near realtime info you would have to hit it 86,400 a day just for that one feed.

There are some API keys owned by warring tags that hit the news feed 86400 times per day. I have no issue with that because they actually use the data and query at an appropriate frequency for their use case. The problems are the disclosure of private information and the owners of the API keys who query excessively.

Slagpit, I'm not sure what beef you have with me or why you seem to be so irritated.
The clan hosting site that you cryptically brought up is likely EVO, which was developed by Qz back in 2008, almost 17 years ago. All of EVO's integrations with the game were built by him, including news, market, and other tools used by the site. If you have anything specific you'd like me to look at, I would be happy to help, especially in Qz's absence.



It's also irritating to me that instead of taking this to private message, IRC, discord, or the other many mediums we can work together on, you instead call me out on the boards.

My original post mentioned the boot drive was full and that the CPU appears to be pinned whenever the problem occurred, with the database daemon taking most of the resources. It was meant to be helpful and genuine, since I have no knowledge as to how many people are currently working on the issue.



Since Slagpit brought up Evo's game site, Evo's scraper bots crashed beginning Feb/11/25 23:21:04. They were down for a few days, so any load on the game was likely reduced until Feb/13/25 13:58:04. Not sure if that correlates with any reduction of errors or load on your end, but despite the EVO bots being down, it still looks like people were reporting errors on this thread during that window:





Let me know if you need any more info. Again, I'm happy to help where possible.

If an API user hit every feed at the throttled rate it would be 432,288 times in 24 hours. All legitimate hits to the feeds. Tools are meant to be used to the extent they allow.

200,000 hits seems like a non-issue considering.

this ^

I cannot believe that I have to explain this.

Qz is the owner and admin of the game server. He has the right to disclose whatever information publicly that he wants. You are a guest on that server who has some incidental level of access due to some evo2025 involvement. As a guest, you do not have any right to disclose publicly any information about that server. If you have some observations that you'd like to share then send them privately to the game admins.

We are having this conversation in public because you disclosed some information that you shouldn't have and you complained about your post getting deleted instead of taking the hint. I didn't say anything in public until you complained about the deletion.

API users, of which you are one, have a responsibility to read the posts on the API Users forum. There is a post there from 9 days ago asking API users to voluntarily reduce their API calls if possible. You can assist in the current situation by fulfilling your responsibilities as an API user.

Do you drive at 160 miles per hour on the highway just because the speedometer goes that high? Or do you instead drive at a lower speed set by a regulation that allows many different vehicles to safely share the highway?

There are exactly 0 api keys that hit all of the different feeds once per second.

I think it's more of an efficiency thing Slag. I'd like to poll the ranks file as fast as possible to catch networth changes. From there, I can surmise all sorts of information. The most I can poll is once every 5 minutes.



I think the point there is that it is theoretically possible, therefore the infrastructure should technically allow for it it.

I was unaware of the api frequency thread until you pointed it out. Thank you for bringing it to my attention.
I've made changes on my end albeit not Evo's, as that is Qz's personal api key.

https://earthempires.com/...-calls-53411?t=1740172334

So to be clear is it Qz hitting the API 200,000 times or Rasp?

There's no doubt redesigning the arch of this game would make it more efficient & it's too bad this game isn't open source by now. With that said, it's not my site so not my decision just my opinion.

I haven't agreed with Slag on just about everything on these forums; but, I have to agree with him on this - posting non-public endpoints, proxy ips ect., is def going to be a no-go. We all know that as developers.

I don't think Rasp did it on purpose of course - as developers & as fans of the game we like to contribute in any way we can. If we had like another channel exclusively for developers (sorta like the developers API) for where a circle of programmers could discuss code suggestions might be beneficial to both sides.

Oh and Rybka is the best.

I would expect that anyone with admin access to evo's site would be happy to make the change but I guess we're just playing PR games here.

I don't think that Rasp's minor disclosure was malicious in any way but of course I'm going to delete stuff like that when I see it. As Celphi has said it's possible to accidentally disclose stuff that shouldn't be public. The best defense is not to say anything at all.

So to whoever this may concern I’m not having any issues now for the last day or two, bonus is fixed and the game doesn’t freeze. Everything is back to normal.

Ask your boss Qz.

I thought I'd first ask the guy who was "happy to help" but like I said, I guess we're just doing PR spinning here. Great use of time on B&S. If you guys want to be fluffty here then go ahead, it just means that the game admins will be even less likely to read the stuff posted here.

This is being worked on by multiple people. As such, there's no need at this time to leave the thread open.