NextDNS Installer

[maghuro]

https://github.com/nextdns/nextdns

 

[alekos]

Just updated to the latest fw version, and was just checking for an update with the nextdns installer script, there wasn't an update but that's when I noticed when I run the 'NEXTDNS' command, nothing appears.
-sh: NEXTDNS: not found
so I uninstalled nextdns, rebooted router, waited 5mins, reinstalled nextdns and install went fine, but still it cannot find NEXTDNS installed.-
- Info - amtm 3.1.6 FW - RT-AC86U (aarch64) FW-384.17
everything else is fine in amtm (diversion/skynet/ntp/sc)
not sure what is going on?
<edit>

found the issue - this command worked:  sh -c "$(curl -sL https://nextdns.io/install)"
this one didn't:  sh -c 'sh -c "$(curl -sL https://nextdns.io/install)"'
I guess the install commands changed recently on github - everything seems ok now.

 

[XIII]

NextDNS went out of beta today:

https://twitter.com/nextdnsio/status/1257869984006647808

After only 6 days in this month I already used 138 K of the 300 K free requests...

Guess I’m becoming a Pro subscriber today (€19,90/year).

 

[XIII]

Done:

 

[XIII]

@Olivier Poitrey Would this be something you guys could do on NextDNS as well?

https://discourse.pi-hole.net/t/youtube-script-seems-to-be-working-very-well/31316

See post 6000 and up in the Diversion topic on this forum where @thelonelycoder is doing an awesome job of implementing this in his ad-blocking tool.

 

[thelonelycoder]

@Olivier Poitrey Would this be something you guys could do on NextDNS as well?

https://discourse.pi-hole.net/t/youtube-script-seems-to-be-working-very-well/31316

See post 6000 and up in the Diversion topic on this forum where @thelonelycoder is doing an awesome job of implementing this in his ad-blocking tool.

It's been a busy four days for me since I found out about that trick. And now Diversion is the first router based ad-blocker that has a fully automated working solution to block YouTube video ads.
Not even Pi-Hole has that automated setup.
Ha!

 

[Olivier Poitrey]

@Olivier Poitrey Would this be something you guys could do on NextDNS as well?

https://discourse.pi-hole.net/t/youtube-script-seems-to-be-working-very-well/31316

See post 6000 and up in the Diversion topic on this forum where @thelonelycoder is doing an awesome job of implementing this in his ad-blocking tool.

It's been a busy four days for me since I found out about that trick. And now Diversion is the first router based ad-blocker that has a fully automated working solution to block YouTube video ads.
Not even Pi-Hole has that automated setup.
Ha!

I spent some time testing this solution. Like all other similar solutions, it does not work for two reasons:

1. A r#—xxxxxx.googlevideo.com hostname used by Youtube CDN will not accept a different IP, you can test it with curl —resolve. You get 400 responses when pointing those hostnames to the IP of another of those endpoints. The "fix IP" of this script is thus equivalent to just blocking the hostname, it's just more complex…
2. The script will basically block any CDN hostname used to load actual content or ads in the past. There is a chance that some ads will reuse the same domain, but you have an equal (I would even say higher) chance that an actual content will reuse such domain as well. After a few days running this script, you will end up blocking most of the content hostnames, and Youtube will become unusable. It is probable that Youtube rotate those hostnames, so instead, it might just keep doing nothing, and sometime block an ad here or a content there.

You could get the same result as this solution by randomly blocking r*.googlevideo.com. It will sometime block ads, but also sometime block content. As you have more content that ads, you'll block more content than ads.

Keep in mind that Youtube stores ads and content on the same endpoints. If you block a r*.googlevideo.com that was used to serve an ad, you are also blocking all the content that is also served by that endpoint.

That's sad, but there is no solution to block Youtube ads at the DNS level. All solution we tested are placebos. Sorry for the bad news

 

[XIII]

Thank you for your analysis.

I’ll keep an eye on the Diversion forum to see which experiences people post there.

 

[Olivier Poitrey]

Thank you for your analysis.

I’ll keep an eye on the Diversion forum to see which experiences people post there.

Every time such solution pops up, you have people posting positive feedback, finding they see less ads. This is called the placebo effect. Then later you get other people reporting they have issues loading content, which is inevitable with those solutions.

We could add a checkbox on NextDNS saying "Reduce Youtube ads" that does nothing, you will get the same feedback of people feeling it's somewhat effective

 

[grub]

The people in the Pi-hole forum thread and Reddit post are reporting excellent results.
The script does not block IPs or hostnames.

(I wrote the original script based on a hunch I had after analyzing too many network dumps.)

 

[Olivier Poitrey]

The people in the Pi-hole forum thread and Reddit post are reporting excellent results.
The script does not block IPs or hostnames.

(I wrote the original script based on a hunch I had after analyzing too many network dumps.)

By replacing the IP of those hostnames by the one of another, it creates 400 errors. So IT IS equivalent to blocking.

You can do the test yourself. In chrome, take one of those video request in the network tab, right click and copy the curl command. You can execute it and see that you get the video data. Now add the —resolve <hostname of the URL>:443:<fix IP> argument. You will get either a 400 error. This is exactly what this script is doing.

 

[grub]

By replacing the IP of those hostnames by the one of another, it creates 400 errors. So IT IS equivalent to blocking.

You can do the test yourself. In chrome, take one of those video request in the network tab, right click and copy the curl command. You can execute it and see that you get the video data. Now add the —resolve <hostname of the URL>:443:<fix IP> argument. You will get either a 400 error. This is exactly what this script is doing.

Do a tcpdump of the network traffic, you will see the video data still flows when this runs. Now I'm not sure how Google is doing their distribution (likely some Anycast funkiness and tracking to ensure the video flows from whatever IP, it's a black box really and that's a guess) When I first tested this, the query would go out but when the forced IP was returned, the packets kept coming.

Did you actually test the thing out?

 

[Olivier Poitrey]

Do a tcpdump of the network traffic, you will see the video data still flows when this runs. Now I'm not sure how Google is doing their distribution (likely some Anycast funkiness and tracking to ensure the video flows from whatever IP, it's a black box really and that's a guess) When I first tested this, the query would go out but when the forced IP was returned, the packets kept coming.

Did you actually test the thing out?

Yes I tested it, and performed some comparison tests too and tried many different approaches.

There is no black magic with youtube video delivery. It’s pretty standard and it does not use anycast or voodoo tech. It’s pure unicast with signed URLS and their server validate the SNI and host header to make sure it’s not altered. It works like many other video delivery systems.

Note that I have been working in video delivery space for the past 15 years.

 

[grub]

Yes I tested it, and performed some comparison tests too and tried many different approaches.

Did you have fewer ads?

 

[Asad Ali]

By replacing the IP of those hostnames by the one of another, it creates 400 errors. So IT IS equivalent to blocking.

You can do the test yourself. In chrome, take one of those video request in the network tab, right click and copy the curl command. You can execute it and see that you get the video data. Now add the —resolve <hostname of the URL>:443:<fix IP> argument. You will get either a 400 error. This is exactly what this script is doing.

As far as I can see/test this script is not replacing the hostnames IP nor blocking it. That's exactly why a nslookup on one of the GoogleVideo subdomains is a prerequisite for the setup. What this script is doing it simply gives out the fixed local YouTube CDN IP whenever there is a Googlevideo subdomains DNS query thus avoiding the second pre adroll IP DNS lookup.

I can see this breaks YouTube in only one case when the local YouTube CDN edge node changes the IP thus invalidating our forced IP setup.

Note: I haven't tested this thoroughly so my assumption can be wrong, in any case, we'll see how it goes in the next few weeks.

 

[Olivier Poitrey]

Did you have fewer ads?

As I said, it blocks some ads and some contents.

You can see the data from one of my test session of yesterday here: https://docs.google.com/spreadsheets/d/1sm9bgyo32WCHLWIne9j1WgvoFbwleTVWb4ZiuIYLY_U.

The ad/content columns shows the number of requests performed to fetch either ads or contents on each hostname. The two next column is the "after the fact blocking" theoretically performed by the script, as it is learning those hostnames from logs. It's theoretical, because as it's a cron job, so it might not learn them right away and thus certainly block less of those.

As you can see, such solution would block twice as more content than ads.

 

[Olivier Poitrey]

As far as I can see/test this script is not replacing the hostnames IP nor blocking it. That's exactly why a nslookup on one of the GoogleVideo subdomains is a prerequisite for the setup. What this script is doing it simply gives out the fixed local YouTube CDN IP whenever there is a Googlevideo subdomains DNS query thus avoiding the second pre adroll IP DNS lookup.

I can see this breaks YouTube in only one case when the local YouTube CDN edge node changes the IP thus invalidating our forced IP setup.

Note: I haven't tested this thoroughly so my assumption can be wrong, in any case, we'll see how it goes in the next few weeks.

The script is creating a hosts file with all the googlevideo.com hostname found in the logs from you past history, and force them to resolve to a single arbitrary IP got from one of them. So, yes it is replacing the IP of those hostnames by the a fixed IP.

The way those servers are programmed, such manipulation of the IP will make those requests fail, which is equivalent to a blocking. And if it was not blocking, I don't see how it could have any effect on ads, as ads and contents are stored indifferently on those servers.

 

[grub]

As you can see, such solution would block twice as more content than ads.

I guess this is a YMMV situation, because everybody else who sets it up correctly are reporting results similar to mine; no in-video ads and an occassional pre-roll ad.

Mine's been running for two weeks, will have to see how long this goes for.

 

[Asad Ali]

The script is creating a hosts file with all the googlevideo.com hostname found in the logs from you past history, and force them to resolve to a single arbitrary IP got from one of them. So, yes it is replacing the IP of those hostnames by the a fixed IP.

The way those servers are programmed, such manipulation of the IP will make those requests fail, which is equivalent to a blocking. And if it was not blocking, I don't see how it could have any effect on ads, as ads and contents are stored indifferently on those servers.

Well in my case the Arbitrary IP is from the local YouTube Egde node in my ISP and I haven't seen any fail video playbacks so far, the way CDNs work, they are connected to all other nodes over the network and if they don't have the requested video in the cache, they'll fetch it over the network so I still don't see how it'll fail to work even if we're using a single CDN IP.

As far as I can tell, the way this works is YouTube always do a DNS query before the ad roll and that gives a different IP then the original answer we get in the start and this method basically just blocks YouTube from that pre ad roll lookup and forced it to keep playing on the actual content.

 

[Olivier Poitrey]

Well in my case the Arbitrary IP is from the local YouTube Egde node in my ISP and I haven't seen any fail video playbacks so far, the way CDNs work, they are connected to all other nodes over the network and if they don't have the requested video in the cache, they'll fetch it over the network so I still don't see how it'll fail to work even if we're using a single CDN IP.

As far as I can tell, the way this works is YouTube always do a DNS query before the ad roll and that gives a different IP then the original answer we get in the start and this method basically just blocks YouTube from that pre ad roll lookup and forced it to keep playing on the actual content.

I explained it, but you guys are free to believe it works differently. At this point, I think I answered why this won't be added to NextDNS, which was the original question.