Let's talk about the BE kicks

[bamf]

There has been much discussion around the fact that there isn't a dedicated post to this topic, so I thought I would go ahead and start one to let the community know what is going on - as well as to begin a dialog on the subject.  

First and foremost let me say this:  no number of BE "mass kicks" is acceptable to @Paratus or myself.  We consider one in a day to be too many.  Also, I define something to a be a "mass kick" when 20 or more players are kicked from a single server in a 30 second interval with the dreaded "client not responding" message.  With that out of the way, let me give you a little background on the number and scope of the kicks (as I'll call them from here on).  

Here are the number of kicks we have had since 12/9/16 (with the count per server 1-4 in parenthesis to the side):  

9th:  9 (5, 2, 2, 0)
10th: 10 (3, 2, 1, 4)
11th: 11 (4, 1, 2, 4)
12th: 1 (S4)
13th: 1 (S1)
14th: 1 (S1)

The 9th-11th are the weekend, and I think we can all agree that the weekends are where we see more kicks.  I have a theory as to why that is the case, and I'll get to that in a minute, but let's take a look at the numbers.  I hear all the time that the mass kicks happen constantly, and while they are certainly prevalent on the weekends I do tend to think that the perception on the number of kicks greatly outweights the reality.  Now three weeks ago I would agree with you that there were more occurences of the kicks since I can look at just S4 and see that there were 16 kicks in a single Saturday for just that one server.  We have made incremental improvements though, so let's talk about what we have done and what we are continuing to do.  

First of all, we implemented kicks based on network activity.  One thing that we have noticed is that a single lagging player has the ability to affect network connectivity to all players in the game.  The kicks we implemented based on ping have been very helpful in ensuring that player is removed from the server faster - thus allowing network connectivity to continue for the rest of the players on the server.  There are additional network kicks that we can put in place (desync and packet loss for instance), but at this time we are only logging those items so that we can try to find a good balance before implementing new kicks based on those items.  

Second, we have been more aggressively monitoring the memory utilization of the Arma server process.  There is a disconnect in what the in game #monitor command reports as utilized memory and what the OS sees as utilized by Arma.  I've seen this discrepency at times be more than 30%, which can tend to point to a memory leak somewhere in the server executable.  That's not something we can fix, but we can try to minimize the leaks by using various memory allocators (mallocs).  We changed the malloc on S1 on Monday, but we are still seeing memory that does not appear to be released to the OS.  Perhaps that's how Arma is coded (to hold onto the memory that it allocates as it goes), but once you pass ~3GB the game gets desynced for everyone and the BE kicks start to happen.  

Third, we are looking at how we access the database from inside the game.  Those of you familiar with how the backend for a Life server works will likely have heard of extDB before.  That's the layer that sits between Arma and the database.  Typically this role is filled by an Open Database Connectivity (ODBC) driver, and I guess you could call extDB (a makeshift) one for Arma.  Since Arma does not work directly with the ODBC driver provided by the database software manufacturer, you are then left with extDB (or Arma2Net) to connect to the database and keep all your houses, vehicles, items, player data, etc all in order.  This layer also has a malloc that uses memory inside the Arma application.  There is a chance that the memory we are seeing as not used inside the game, but used according to the OS could be leaked from the malloc for extDB.  We are investigating this, as well as looking to update extDB to more recent version.  We have a customized version of extDB, so we will need to redo those customizations as we upgrade that as well.  

Finally, we continue to analyze the logs we have to see what correlations we can find inside them.  Here is a summary of those findings:  

1. The kicks are more frequent when the servers are full, but the servers being full does not indicate that a kick is about to (or even likely) to happen.  
2. The kicks tend to happen more when the OS sees ~3GB of RAM being used by the Arma server application.  Some of this is anecdotal since I need to be made aware of the kick and then check the RAM utilization inside of the OS.  I will say this though, this last weekend I checked the RAM on the server processes when I could, and once it got high I asked the admins to hard restart the server.  I think that did make a difference in the overall number of kicks for the weekend, but that's not a tenable solution to go forward.  
3. The kicks tend to happen more on the weekend, which is when the servers are not only full - but also have higher turnover resulting in more unique users (UUs as @Gnashes likes to say) on the servers throughout those days.  As more players come and go to the server, more RAM is utilized by the game.  The servers are full during prime time on weeknights, but we just don't see the same number of kicks during the week.  We do have have fewer UUs during the week as well.  

Given all of that, I believe the Arma server process starts to lag once the RAM utilization gets above some unknown threshold.  The game and OS allocate memory as players come and go, as well as during the normal course of a server session.  As the server begins to lag, the network communication begins to lag as well - and at some point we get players receiving "client not responding" kicks.  

Some of you may have seen that 64-bit versions of the game hit the development branch of Arma today.  I am hopeful that a 64-bit server version will alleviate the issues of RAM constraints and allow the server to not lag (and therefor not kick players).  Of course, all of this is still in development and could be subject to other bugs and issues.  Further, we will have to make sure that all the external DLLs we use with the Arma server are updated to 64-bit and free of bugs as well, but having a 64-bit server will finally allow us to utilize more of the horsepower we have on each and every server that we have in the data center.  

I'll talk to Paratus in the morning, but perhaps we can be a tad more aggressive with the hard restarts we do.  Currently we do a hard restart after 3 server sessions, but perhaps we should move that to after 2 server sessions.  We will also continue to monitor the logs, RAM utilization, and player counts to see if we can try to really narrow down when we get into situations where the kicks begin to happen.  

I hope this was informative for you all, and I'm happy to hear your feedback and suggestions.  

Bamf

P.S.  I'm going to ask that everyone stay on topic and not turn this into a flaming or finger pointing thread.  Posts that are off topic or just generally unhelpful will be heavily moderated so that we can keep this as an open discussion with the community.  

Edit:  One final thing of note is that this does happen in other communities as well, so it is not isolated to just Asylum.  

[Crossfade]

Thanks bamf..

[JIMBO]

There we go, some communication, progress!

 

now can I be removed off of modqueue for trying to get community managers to bring it up, thanks

[Sylus Degalle]

I don't know what any of that means, But goodluck on fixing it. I believe.

[Mason]

Thank you! Finally we have some answers. I was getting tired of the standard "It's Bohemia's fault" line we have been fed for the past few weeks with no elaboration what so ever. Just needed more communication.... I am glad to see that there is work on this issue. Thanks BAMF!

Edited December 15, 2016 by Biground

[J. Urbanowski]

Thank you for the update.

[ρhantom]

Thank you.

[Boris]

Boris approves of community communication.

[explicit]

This is what we needed. Thank you.

[[df]Smokahontas]

"Olympus seems fine" - Diseased25

[DreamC]

I timed out halfway through reading this, but good job.

[Guest]

22 minutes ago, Jimbo said:

There we go, some communication, progress!

 

now can I be removed off of modqueue for trying to get community managers to bring it up, thanks

This may shock you; but the Community Managers have zero knowledge or ability to do Development related things.

You know, all of this stuff.

[JIMBO]

Just now, Gnashes said:

This may shock you; but the Community Managers have zero knowledge or ability to do Development related things.

You know, all of this stuff.

That's not what I nagged them for, I nagged them to speak to the community, to let us know something is being done, if we aren't told that you guys are trying to fix it then we can only assume nothing is being done.

You know, 8 months of this stuff.

[Sylus Degalle]

3 minutes ago, Reapered said:

I'm typically retarded when it comes to the subject of stuff like this, but is it possible in anyway that the mass kicks usually happen around full server times because a lot of fighting typically happens during those times. Like could ifrit smokes and constant fighting *overload* the servers in any way. plz dont roast if this is a stupid question 

He said that it happens more on peak time, Like the weekends.

[Mahdizzle]

Good very good now next step is to get the CMs to do this and ofc inform them so they can 

One small step for @bamf one giant leap for @Paratus 

[Furnie Mack]

Thanks for this post @bamf 

[.Sean]

Finally some transparency first bit I've seen with the devs and such since I came back in April thanks guys!

[Mr Smirnoff]

@bamf so lets say you have an unlimited budget and can get the best server possible would that help this in any way? If you get the server with the most ram possible and best CPU would performance improve or its not going to make a difference? 

[Hanzo/Dirty Scrubz]

13 minutes ago, Mr Smirnoff said:

@bamf so lets say you have an unlimited budget and can get the best server possible would that help this in any way? If you get the server with the most ram possible and best CPU would performance improve or its not going to make a difference? 

No it wouldn't because the ARMA server is 32-bit at the moment with a max allocation of 2 GB. BI is working to update the 32-bit portion to 3 GB while the 64-bit server will essentially remove the aforementioned limitation entirely. So in short, they aren't even using the ram they have right now.

[Mr Smirnoff]

Just now, Hanzo/Dirty Scrubz said:

No it wouldn't because the ARMA server is 32-bit at the moment with a max allocation of 2 GB. BI is working to update the 32-bit portion to 3 GB while the 64-bit server will essentially remove the aforementioned limitation entirely. So in short, they aren't even using the ram they have right now.

Oh I see servers sitting at around 2gb ram all the time pretty much. 

[Hanzo/Dirty Scrubz]

2 minutes ago, Mr Smirnoff said:

Oh I see servers sitting at around 2gb ram all the time pretty much. 

Pretty much and as @bamf explained its why the kicks happen, once the 2GB threshold is crossed. BI addressed this as well here: https://dev.arma3.com/post/oprep-64-bit-executables

Quote

Speaking of memory, let us translate these huge numbers to something more understandable. In the vocabulary of computer memory, 2^32 can be translated to 4 gigabytes while 2^64 equals to 65 petabytes. In simplistic terms, a 32-bit system can physically address at most 4 GB of memory. On top of that, the address space of a 32-bit application running on such a system is usually split in two parts. On Windows, each of them is 2 GB. One is reserved for your applications, the other one for Windows itself.

 

An important side effect of having larger address space is that we expect fewer memory-related crashes to happen. With no free memory, sooner or later the game (or any other application for that matter) is bound to reach a state where it simply has no other choice but to cause us all sorrow. Memory limits are set in stone; whether a crash is caused by a GPU driver or Arma itself does not really matter from a player's point of view. Trust us, there is nothing more painful then seeing you sad because the game crashed. In 64-bit Arma, this should be less likely.

 

However, although it is true the support for 64-bit has yet to hit our Main Branch, we decided to give you all at least something as a small appetizer. Starting with Update 1.66, when run on 64-bit Windows, 32-bit Arma 3 will be able to effectively work with more than just 2 GB of memory. The current limit has been increased to 3 GB. Magic, yay!

 

Edited December 15, 2016 by Hanzo/Dirty Scrubz

[Legit]

4 hours ago, Gnashes said:

This may shock you; but the Community Managers have zero knowledge or ability to do Development related things.

You know, all of this stuff.

You shouldn't use semicolons with conjunctions. I can't take your mildly aggressive posts seriously when you insist on misusing a glorious form of punctuation. I just don't want typos in new features

But I digress, thank you for the communication and insight into the problem. 

[Mr Smirnoff]

29 minutes ago, Gnashes said:

See:

 

Oh so checking the #monitor values is pretty much pointless to diagnose anything oh well. 

[Henry Facesmasher]

Cool. What about the mem leaks? Think they go hand in hand? I play on multiple arma servers and I do not experience mass BE kicks or mem leaks what so ever. I do understand that life servers tend to be heavily modded, still shouldn't be getting them.

[Hanzo/Dirty Scrubz]

25 minutes ago, Henry Facesmasher said:

Cool. What about the mem leaks? Think they go hand in hand? I play on multiple arma servers and I do not experience mass BE kicks or mem leaks what so ever. I do understand that life servers tend to be heavily modded, still shouldn't be getting them.

 

[Seán That Irish Guy]

30 minutes ago, Gnashes said:

I'm amazed Haych hasn't been in here yet. He's been the most vocal person about the BE Kicks.

@Haych; where you at brother?

 

ewwwwww Gnashes called you brother.... @Haych

[Haych]

On 16/12/2016 at 2:34 PM, Gnashes said:

I'm amazed Haych hasn't been in here yet. He's been the most vocal person about the BE Kicks.

@Haych; where you at brother?

 

 

 

I'm pretty much done being vocal about it because i've been going on about it for the last 7-8 months. At this point, i'm just hoping it gets fixed so I can start playing Asylum again. My only worry is, what if the 64 Bit and other methods mentioned doesn't resolve the issue, what's going to be next? Lets just hope it does.

Nice to see that you guys are being more vocal about the matter, it was long due and hopefully it gets better from now on. Don't forget @bamf, you have 2 CM's who would probably love to know more about server issues and such so don't keep them in the dark, they can't do their job if they aren't being told anything.

^{edit by devran: i dont know why haych cares so much about asylum the retard doesn't even play he's supposed to be playing overwatch but he's on here typing out long paragraphs going into detail about BE kicks like it even effects him, ive found that hes increasingly getting more and more retarded can the admins ban him on the forums i need the old haych back.}

Edited December 19, 2016 by Haych

[Vanilla Coke]

Now, with that out of the way, is there any reason for the ridiculous amount of mem leaks I've been experiencing in the last week or two? I have a feeling that it's probably a Bohemia thing, as it started happening after their last patch, but maybe you guys know something more? Just looking for some insight.

[signal]

Well, instead of trying to revive tanoa there could possibly be another altis server set in place with a lower server population (80-90) and seeing if that opens up and relieves some of the BE kicks. This would also reopen up housing to many people trying to purchase exclusive homes. @bamf

[Tyreck]

5 minutes ago, signal said:

Well, instead of trying to revive tanoa there could possibly be another altis server set in place with a lower server population (80-90) and seeing if that opens up and relieves some of the BE kicks. This would also reopen up housing to many people trying to purchase exclusive homes. @bamf

Even if this isn't a permanent solution. +1 

[Henry Facesmasher]

6 hours ago, Will said:

Memory leaks are associated to this yes. But is usually down to a coding error. The memory crashes that happen every so often are to my knowledge an effect from the recent patch. Because of the memory issues we have with allocation this then causes it to be more noticable on life servers especially because of the amount of variables life servers use.

I posted a reply to someone else about the memory leaks here which goes into more detail.

 

I had memory leaks before the newest patch. But now I can tell when i'm going to get them. It happens after about 1 1/2 hour of game play i'll usually get a fake mem leak after the first 45 min, then it'll go into another fake one after the second one it'll do the real one.

[Eli]

2 hours ago, Vanilla Coke said:

Now, with that out of the way, is there any reason for the ridiculous amount of mem leaks I've been experiencing in the last week or two? I have a feeling that it's probably a Bohemia thing, as it started happening after their last patch, but maybe you guys know something more? Just looking for some insight.

It is a Bohemia thing, been playing on other servers and I kept on getting them. I updated my drivers, updated my performance binary, and changed my parameters to -malloc=jemalloc_bi. Haven't had a mem leak in like a week now.

[Dork]

42 minutes ago, Will said:

If you can pin point when they happen like if it occurs always when you exit a vehicle say as an example it would be very useful in identifying the source

i dont get them that much like everyone else, but normally if i exit a vehicle and start shooting or if someone starts shooting towards me seems to be the most times that i get them.

[BioHazard]

Good to hear!  

[Otis Jenkins]

i love you guys so much i come back to read the forums and some real good is happening YALL SOME GOOD PEOPLE

[Haych]

@bamf @Olio I recommend putting this announcement on a banner on the front page if you want it to have more visibility for the whole community to see, you lot should use that feature on IPBoard more often. If no one bumps it, no one reads it like most announcement posts because they get buried quick as Change Logs are always more popular. Montages get more views than important announcement posts for this reason , +1 for more visibility.

for example 

[henky]

On 16-12-2016 at 8:31 PM, Vanilla Coke said:

Now, with that out of the way, is there any reason for the ridiculous amount of mem leaks I've been experiencing in the last week or two? I have a feeling that it's probably a Bohemia thing, as it started happening after their last patch, but maybe you guys know something more? Just looking for some insight.

same thing to me, yesterday was insane for the 4 hours i played i had atleast 4 memleaks

[HotWings]

On 12/16/2016 at 1:31 PM, Vanilla Coke said:

Now, with that out of the way, is there any reason for the ridiculous amount of mem leaks I've been experiencing in the last week or two? I have a feeling that it's probably a Bohemia thing, as it started happening after their last patch, but maybe you guys know something more? Just looking for some insight.

I was watching a stream of an Olympus dev last night and he said its very much an issue there as well. They just call it a texture bug over there.

[.Sean]

Just now, Gnashes said:

Arma tweaked one of the memory allocators last patch; my guess is they redid something to be more optimized on it; and it shit the bed with other allocators.

It's definitely stock Arma. Memory leaks on the client result from client issues. AUS was no different; the issue there was in a clientside mod.

Do you know of any memory allocators we can use to prevent from mem leaking because I looked at a police hunter wrong?

[signal]

57 minutes ago, House. said:

Do you know of any memory allocators we can use to prevent from mem leaking because I looked at a police hunter wrong?

I was getting more memleaks with the default arma 3. Switch to one of the latest performance builds I've gotten 2 mem leaks in the past week rather than 4-5. Also fixes any FPS drops