[Release] FreshJR Adaptive QOS (Improvements / Custom Rules / and Inner workings)

[gmangt4]

Activision say these ports are used for Blops 3 on PS4

• TCP: 80, 443, 1935, 3480
• UDP:
• TCP and UDP: 3478-3479, 3074, 3075

Hope that helps and is not oo off topic

 

[Vexira]

@Vexira @FreshJR
I welcome you to check... As I don't PC game these days. I however have gamed a good amount on PS4, and have played all the COD games. But I will say I barely played COD IW, as I didn't like the new one. I don't even currently have it installed on my PS4. So I can't 100% confirm it uses port 3074 to pass gaming packet's once in a lobby. But I can say all previous COD games on PS4, I have personally seen them use port 3074 to pass active match data. I do however think in the past they have listed a port range of like 3074-3076, for people who have multiple PS4's online at same time playing the same game.

With that said, from what I have seen, a lot of the time. Even with two/three PS4 online under the same router playing the same COD game online, they all still try pushing data threw the same 3074 port, instead of one using 3074, other using 3075, etc.. Reason often times you will have people with these types of setup's. Posting online how one system is stating a OPEN nat type, while the second/third console saying Moderate nat type. Because the router itself is opening 3074 via UPnP, or Port Forwarding for the first console playing the game. Then the Second/Third console also request the same port to be open, and is declined, as the first system already has it open. So them system's report back a moderate nat type, instead of requesting say 3075/3076, and they still end up pushing data threw the same 3074 port.

Again with all this, This is talking strictly COD games for PS4 for me. As I stated in a before post, I also played Battlefield Hardline, on my PS4, and it was using port 3659 to pass active game data while in a match. After watching over my PS4 connection the last 2-3 days. A lot of general PS4 traffic, and download's are passing data between ports in the 49k to 65k range. One other thing PS4's party chat, normally passing voice traffic over port 9307.. Which I seen the last couple days while in party chats. But I do know PS4 list I think between 9304-9309 as ports voice chat will use, depending on who you're in voice chat with, and if you have multiple PS4's online at the same time, using voice party chat.

The other console not having open nat means that, the client is using or requesting port 3074 but not receiving it and the router is not preforming port translation correctly, if you observe upnp logs with two sets of eg bops 3 open you will see what I mean one will get the primary port the other will get primary on inbound and a different on external, if I remember correctly, there's a good thread here called multiple xbox ones, I posted a few things, though I will hopefully get my guide on open nat written to see if I can't help anyone. Also cod on pc is infested like many other pc shooters with cheaters thus killing the player base, also the game is killed by supply drops, I do play a lot of xbox, but the sad thing is my head set is not fully supported by the console. I find it it easier to get a match in Overwatch than in cod on pc.
Though Merlin squash most of the major upnp issues, there's are still a few yet to be fixed but multi console support works for the most part thanks to Merlins hard work.
Also ill assume that you have been watching the upnp log for ports.

 

[lilstone87]

Also ill assume that you have been watching the upnp log for ports.

@Vexira @FreshJR
Actually I did still have a older router of mine laying around, which supported Tomato firmware. So I loaded it on it, and ran my PS4 threw it. Which if you have used Tomato before, you know the QoS system actually has a section in which it shows all active addresses, and data passing threw different ports. So last couple days, I have been monitoring my PS4 while using it that way.

So I was able to see which port's were active, and which of them was passing active data while certain things were in use... Like using party chat, and playing both Call of Duty Black Ops 3, and Battlefield Hardline online multiplayer. Between the three things I used. Port 9307 was used for party chat primarily, I do think I once seen port 9305 used while I was in a party chat with 7 people. However 99.9% of the time port 9307 was used for most people I was connected to in party chat. Which party chat on PS4 is setup P2P, not any kind of server hosting the chat.

Also like I mentioned while playing Battlefield Hardline, while in a game/lobby port 3659 is used to pass gaming data. I know these game list multiple ports they use, and they might use them to log certain stuff like in game voip, if you're not using party chat, and statistic data for the matches you play. However most these games at least for console games I have played.. Which I have been a console gamer online for a long time now. Strictly uses one port to pass active gaming packet's while in a match. Battlefield Hardline is port 3659, Black Ops 3 is port 3074, and likely all the COD games on PS4 use port 3074 to pass in match gaming packet's. Which I know a lot of people still actively play Call of Duty on console, but for me the last couple games have went downhill for me. So I haven't played COD as much myself... But hope the new COD WWII brings some enjoyment back to COD series, with things being boots on the ground again.

 

[lilstone87]

@FreshJR
Well I turned on my PS4 a little bit ago, and one of my games started downloading a patch, and after it finished. I decided to take a screenshot of Tomato's QoS View Detail's page. Just to show you what the PS4 has open, in mostly a idle state.. As I just powered it on, and let it sit till after the patch download was done. No game started, or anything. So I will attach a screenshot in this post, which you might be able to use, to see what the PS4 is accessing, before any games are involved.

I know I mentioned seeing the PS4 use between 49k to 65k port range. However them port's are listed as source ports for the PS4, yet for the destination port(Sony's servers, and such.). The destination port is a mixture of port 80/443.

 

[lilstone87]

Also here's a second screenshot of my PS4 active connection from just a few minutes ago. This is after like a hour of me playing Battlefield Hardline multiplayer. I had tomato set to show active connection's with the most used data at the top. You can see Hardline was at the top using source port of 3659.

 

[Vexira]

@Vexira @FreshJR
Actually I did still have a older router of mine laying around, which supported Tomato firmware. So I loaded it on it, and ran my PS4 threw it. Which if you have used Tomato before, you know the QoS system actually has a section in which it shows all active addresses, and data passing threw different ports. So last couple days, I have been monitoring my PS4 while using it that way.

So I was able to see which port's were active, and which of them was passing active data while certain things were in use... Like using party chat, and playing both Call of Duty Black Ops 3, and Battlefield Hardline online multiplayer. Between the three things I used. Port 9307 was used for party chat primarily, I do think I once seen port 9305 used while I was in a party chat with 7 people. However 99.9% of the time port 9307 was used for most people I was connected to in party chat. Which party chat on PS4 is setup P2P, not any kind of server hosting the chat.

Also like I mentioned while playing Battlefield Hardline, while in a game/lobby port 3659 is used to pass gaming data. I know these game list multiple ports they use, and they might use them to log certain stuff like in game voip, if you're not using party chat, and statistic data for the matches you play. However most these games at least for console games I have played.. Which I have been a console gamer online for a long time now. Strictly uses one port to pass active gaming packet's while in a match. Battlefield Hardline is port 3659, Black Ops 3 is port 3074, and likely all the COD games on PS4 use port 3074 to pass in match gaming packet's. Which I know a lot of people still actively play Call of Duty on console, but for me the last couple games have went downhill for me. So I haven't played COD as much myself... But hope the new COD WWII brings some enjoyment back to COD series, with things being boots on the ground again.

ahh i see also long as the port number is on both inbound and outbound that the main port number the game client uses. Though it appears that on pc infinte warfare uses 3074 like the rest of the games, still to this day i dont understand why trearch changed the black ops 3 pc port from 3074 to 27017 it made all the port forward guides for the pc version invalid. But still props for your efforts, im starting to belove that some games may use slightly diffrent ports on pc compared to consoles.

 

[FreshJR]

That's a damn nice traffic analyzer.

We have nice one too, with a a nicer interface I may add, but instead it only sorts by device then application name. Wish we could append port and ip usuage into our traffic statistics. Most likely this was all intended to be automatic to be all handled with definitions so we wouldn't have to mess with and have it be confusing for many customers. Remember, we are working outside the box here.

Either way, without a more advanced analyzer, doesn't Sony or the game developers themselves publish this information online?

Check out Apple, they did documentation correctly, https://support.apple.com/en-us/HT202944

Even T-Mobile has this information published on their website.

Looking up Linux traffic analyzers, many nice ones are available to be installed. You can also use wireshark from pc to sniff all traffic. Many options

 

[MarCoMLXXV]

Hi @FreshJR , first of all thanks for your effort to write and maintain your script. I installed it a few days ago and everything worked flawless. Yesterday I did a speedtest and the results were much lower than usual, so I thought of your script. I noticed the speedtest was registered under File Transferring, so I edited the variables so that File Transfers was given more bandwidth and lowered some other. Checked that for both upload and download the percentages were still 100% in total each. Rebooted my router, did the same speedtest, results were still unsatisfying. I checked syslog and saw a huge amount of log lines like below:

Jul 28 19:52:34 dMP17 kernel: TCP: time wait bucket table overflow
Jul 28 19:52:34 dMP17 kernel: TCP: time wait bucket table overflow
Jul 28 19:52:34 dMP17 kernel: TCP: time wait bucket table overflow
Jul 28 19:52:34 dMP17 kernel: TCP: time wait bucket table overflow
Jul 28 19:52:34 dMP17 kernel: TCP: time wait bucket table overflow
Jul 28 19:52:34 dMP17 kernel: TCP: time wait bucket table overflow
Jul 28 19:52:34 dMP17 kernel: TCP: time wait bucket table overflow
Jul 28 19:52:34 dMP17 kernel: TCP: time wait bucket table overflow
Jul 28 19:52:34 dMP17 kernel: TCP: time wait bucket table overflow
Jul 28 19:52:34 dMP17 kernel: TCP: time wait bucket table overflow
Jul 28 19:53:04 dMP17 kernel: net_ratelimit: 79 callbacks suppressed
Jul 28 19:53:04 dMP17 kernel: TCP: time wait bucket table overflow
Jul 28 19:53:04 dMP17 kernel: TCP: time wait bucket table overflow
Jul 28 19:53:04 dMP17 kernel: TCP: time wait bucket table overflow
Jul 28 19:53:04 dMP17 kernel: TCP: time wait bucket table overflow
Jul 28 19:53:05 dMP17 kernel: TCP: time wait bucket table overflow
Jul 28 19:53:05 dMP17 kernel: TCP: time wait bucket table overflow
Jul 28 19:53:05 dMP17 kernel: TCP: time wait bucket table overflow
Jul 28 19:53:05 dMP17 kernel: TCP: time wait bucket table overflow
Jul 28 19:53:05 dMP17 kernel: TCP: time wait bucket table overflow
Jul 28 19:53:05 dMP17 kernel: TCP: time wait bucket table overflow

There were literally thousand of these lines, which I've never seen before in syslog. Then, a short time later, I noticed:

Jul 28 19:54:34 dMP17 marco: Adaptive QOS: Modification Script Started
Jul 28 19:55:06 dMP17 marco: Adaptive QOS: No change required for Unidentified Traffic Container or Custom Rules
Jul 28 19:55:07 dMP17 marco: Adaptive QOS: Changing minimum alloted bandwidth per QOS category to user defined percentages
Jul 28 19:55:38 dMP17 marco: Adaptive QOS: Modification Script Started
Jul 28 19:56:01 dMP17 marco: Adaptive QOS: Modification Script Started
Jul 28 19:56:13 dMP17 kernel: SysRq : Emergency Sync <----------
Jul 28 19:56:13 dMP17 kernel: Emergency Sync complete <---------
Jul 28 19:56:15 dMP17 kernel: mod epilog takes 0 jiffies
Jul 28 19:56:15 dMP17 kernel: IDPfw: Exit IDPfw
Jul 28 19:56:15 dMP17 kernel: Stop the IPS/AppID engine...
Jul 28 19:56:16 dMP17 kernel: IDPfw: Exit chrdev /dev/idpfw with major 191
Jul 28 19:56:16 dMP17 kernel: [udb_exit:959] Free udb at cde1f000
Jul 28 19:56:16 dMP17 kernel: [udb_exit:960] Free app at d2db3000
Jul 28 19:56:16 dMP17 kernel: [udb_exit:961] Free patrol table at d2dd6000
Jul 28 19:56:16 dMP17 custom_script: Running /jffs/scripts/services-stop
Jul 28 19:56:16 dMP17 WEBDAV_Server: daemon is stopped
Jul 28 19:56:16 dMP17 NAT_Tunnel: AAE Service is stopped
Jul 28 19:56:16 dMP17 NAT_Tunnel: AAE Service is stopped
Jul 28 19:56:16 dMP17 iTunes: daemon is stopped
Jul 28 19:56:16 dMP17 FTP_Server: daemon is stopped
Jul 28 19:56:17 dMP17 pixelserv: 3147 uts, 0 log, 211 req, 432 avg, 538 rmx, 153 tav, 9997 tmx, 193 slh, 2 slm, 0 sle, 9 slu, 1 nfe, 0 gif, 0 ico, 6 txt, 0 jpg, 0 png, 0 swf, 0 sta, 0 stt, 0 ufe, 0 rdr, 0 nou, 0 pth, 0 204, 0 pst, 0 hed, 0 bad, 0 err, 0 tmo, 204 cls
Jul 28 19:56:17 dMP17 pixelserv: exit on SIGTERM
Jul 28 19:56:18 dMP17 Samba_Server: smb daemon is stopped
Jul 28 19:56:18 dMP17 kernel: gro disabled
Jul 28 19:56:18 dMP17 Timemachine: daemon is stopped
Jul 28 19:56:18 dMP17 disk_monitor: Finish
Jul 28 19:56:19 dMP17 ntpd: ntpd exiting on signal 15 (Terminated)

My initial changes (before things went wrong) were:

    ####################  Variables Setup #####################

    #Percent of download speed guaranteed per QOS catagory, change below as desired (sum should equal 100)
        NetControl_DownBandPercent=5                    #This value can be adjust as desired
        VoIP_DownBandPercent=5                            #This value can be adjust as desired
        Gaming_DownBandPercent=15                        #This value can be adjust as desired
        Others_DownBandPercent=10                        #This value can be adjust as desired        #Note: New destination for all unidentified traffic per script default
        WebSurfing_DownBandPercent=20                    #This value can be adjust as desired
        Video_DownBandPercent=35                        #This value can be adjust as desired
        FileTransfer_DownBandPercent=5                    #This value can be adjust as desired
        Default_DownBandPercent=5                        #This value can be adjust as desired        #Note: Original destination all for unidentified traffic, no traffic should flow here
  
    #Percent of upload speed guaranteed per QOS catagory, change below as desired (sum should equal 100)
        NetControl_UpBandPercent=5                        #This value can be adjust as desired
        VoIP_UpBandPercent=5                            #This value can be adjust as desired
        Gaming_UpBandPercent=15                            #This value can be adjust as desired
        Others_UpBandPercent=30                            #This value can be adjust as desired        #Note: New destination for all unidentified traffic per script default
        WebSurfing_UpBandPercent=15                        #This value can be adjust as desired
        Video_UpBandPercent=20                            #This value can be adjust as desired
        FileTransfer_UpBandPercent=5                    #This value can be adjust as desired
        Default_UpBandPercent=5                            #This value can be adjust as desired        #Note: Original destination all for unidentified traffic, no traffic should flow here

and after the changes I made, it looked like:

####################  Variables Setup #####################

    #Percent of download speed guaranteed per QOS catagory, change below as desired (sum should equal 100)
        NetControl_DownBandPercent=5                    #This value can be adjust as desired
        VoIP_DownBandPercent=5                            #This value can be adjust as desired
        Gaming_DownBandPercent=15                        #This value can be adjust as desired
        Others_DownBandPercent=5                        #This value can be adjust as desired        #Note: New destination for all unidentified traffic per script default
        WebSurfing_DownBandPercent=20                    #This value can be adjust as desired
        Video_DownBandPercent=30                        #This value can be adjust as desired
        FileTransfer_DownBandPercent=15                    #This value can be adjust as desired
        Default_DownBandPercent=5                        #This value can be adjust as desired        #Note: Original destination all for unidentified traffic, no traffic should flow here
  
    #Percent of upload speed guaranteed per QOS catagory, change below as desired (sum should equal 100)
        NetControl_UpBandPercent=5                        #This value can be adjust as desired
        VoIP_UpBandPercent=5                            #This value can be adjust as desired
        Gaming_UpBandPercent=15                            #This value can be adjust as desired
        Others_UpBandPercent=25                            #This value can be adjust as desired        #Note: New destination for all unidentified traffic per script default
        WebSurfing_UpBandPercent=15                        #This value can be adjust as desired
        Video_UpBandPercent=15                            #This value can be adjust as desired
        FileTransfer_UpBandPercent=15                    #This value can be adjust as desired
        Default_UpBandPercent=5                            #This value can be adjust as desired        #Note: Original destination all for unidentified traffic, no traffic should flow here

When I saw the kernel doing an emergency sync, I decided to remove your script for now. No more lines as quoted above anymore, and through the night it remained stable and syslog hasn't showed any unknown errors anymore.

Any suggestions to what has caused this? Thanks in advance.

 

[FreshJR]

@MarCoMLXXV send me a copy of the exact file you uploaded to the router.

I haven't run into the issue myself

 

[MarCoMLXXV]

@MarCoMLXXV send me a copy of the exact file you uploaded to the router.

I haven't run into the issue myself

Thanks for your reply. I've posted mine here: https://pastebin.com/WRR0dGsU

The only thing I changed were the values in the section 'variables setup', the rest of the script remained untouched. Maybe a wrong distribution of the values across containers?

 

[FreshJR]

Thanks for your reply. I've posted mine here: https://pastebin.com/WRR0dGsU

The only thing I changed were the values in the section 'variables setup', the rest of the script remained untouched. Maybe a wrong distribution of the values across containers?

I need the actual text file, not the pastebin, PM it to me.

When I downloaded ur pastebin, it had the WRONG return line character. If that indeed is the case with the file you uploaded to the router, its not compatible and will lead to weird problems. I warned about this in the initial post.

Good on not adding spaces after the = operator. That is because variable assignment is very picky aswell in shell scripts. Everything else was unchanged as you said.

 

[MarCoMLXXV]

I don't see a way to attach a file through PM? There's no 'Upload file' button in Conversations? Or am I missing something.

As for the line endings. I always edit in NotePad++ and make sure it's EOLs are converted to UNIX (LF) before I upload anything to my router. Just checked, it's definitely UNIX LF. Got a nine year old around here so messing up my router means all hell will break loose here

Does this help: https://www.dropbox.com/sh/e3v7z22h1l1r3gq/AACyF2pFJs5WzayI6nEdkLtFa?dl=0 ? I've attached it to this post as well.

I don't get why Pastebin would convert the line feeds, though.

Strange thing is, it worked perfectly at first, untill I made some changes to the variables, then things went down the hill... Even after a reboot, the errors were still flooding my logs.

 

[Nigel Jones]

I've noticed on a few occasions now that I see my speedtest.net result a little slow (150-160 Mbps) on my PC compared to usual (215-220 Mbps).

Few systems are connected. I see little data flow. I disable QOS -> full speed. I reenable QOS (which brings in the adjustments) -> also full speed.

Any ideas? This could be a merlin QOS issue, or an issue with these rules, perhaps a memory leak in the kernel too - any tips on debugging welcome?
If it continues to occur I'll first try disabling the mods here...

AC-3200 running .67 & script 1.9

 

[Uthall]

Hello,

I want to add port 25461 into the script and have it classified as "Video and Audio Streaming"

How do i do that?

Thanks
Jeff

EDIT: Its OK, i think i worked it out

 

[MarioCaires]

Hello,

I want to add port 25461 into the script and have it classified as "Video and Audio Streaming"

How do i do that?

Thanks
Jeff

EDIT: Its OK, i think i worked it out

Ahahah... I think I have the same "Streaming" service as you! However I found that the traffic is already correctly identified even w/o the custom rule.

 

[Uthall]

Lol....

Well i wish mine was, what version of the Merlin firmware are you using, and i take it your not using the custom script?

 

[MarioCaires]

Merlin's version is in my signature and I'm using the script with some custom rules, mostly for torrent traffic.

 

[Uthall]

Question, the variables in the script don't match the names in the QOS Statistics window

eg.
script = "Streaming"
QOS Statistics = "Video and Audio Streaming"

Does that matter?

 

[MarioCaires]

Question, the variables in the script don't match the names in the QOS Statistics window

eg.
script = "Streaming"
QOS Statistics = "Video and Audio Streaming"

Does that matter?

Doesn't matter, they're the same, just named differently by FreshJR.

 

[FreshJR]

Defiantly different names than webUI. Spaces in variables are a no-no. Valid names are the following (Case Sensitive)

VOIP
Gaming
Others
Web
Downloads
Default