Cloud9 DNS

[unclebuk]

https://en.wiktionary.org/wiki/kiboze

Other things to do, yes, but Quad9 is a high priority for us, and heck, I'm just sitting around an airport lounge waiting for my next plane, so I might as well be useful, right?



Sure.



Relative to those two, the answer is the same: privacy and security.

A lot of people look at recursive DNS and think that performance is the thing that matters, because it's the thing that they can measure. Performance is easy to see, because anybody can run dnsperftest to see which gives the quickest average response time from their location. But of the four large ones (OpenDNS/Umbrella being the fourth) all are likely to give you very good performance if you're in North America or Western Europe. Because the other three are commercial, they focus their effort in the places where people have the most money to spend, so you're less likely to get good performance from them if you're in Africa or South America or the Caribbean or South Asia, for instance. But if you're in the US, or Canada, or France, or Germany, any of the four will give you perfectly sufficient performance, and no amount of tinkering or switching is likely to yield any user-noticeable improvement. But performance isn't the point. Google was already there when we set up Quad9, and we're not going to blow our donors' money solely to one-up somebody's commercial offering on the basis of performance.

The point was privacy and security.

Google and Cloudflare make money collecting and selling personal information. Whatever you may think about the morality of that, it's flat-out illegal in Europe, and Quad9 was started because European privacy regulators asked us (meaning PCH, in this case) to stand up a GDPR-compliant recursive resolver, as an existence-proof that it was possible to run this critical infrastructure without paying for it by stealing users' personal information and hawking it to data-brokers.

So, unlike the others, Quad9 does not collect personal information. Quad9 does not have a concept of a "user" to hang records off of, and does not collect any IP addresses. Quad9 is the only big anycast resolver that doesn't collect personal information, and it's the only free one that's GDPR-compliant. (Cisco's commercial Umbrella offering is GDPR-compliant and doesn't sell information.) There are people who say that it's okay to collect information if you don't do anything bad with it, but that's completely wrong, because breaches happen all the time. Any information you collect will eventually be stolen, and when it's stolen, it'll be sold. So, don't collect unnecessary information in the first place.

Relative to security, malware and phishing and so forth are a horrible problem, particularly with IoT junk. Botnets are getting very large, and the DDoS attacks they source are a vast problem. So using the recursive resolver to block contact between bot software and its C&C, as David did with OpenDNS, is an excellent way to protect users from malware, and to protect the Internet from infected devices. Whereas OpenDNS has Cisco as its sole source of "threat intelligence," Quad9, as a not-for-profit Internet industry project, has twenty, including Cisco and IBM and F-Secure and many others. So Quad9 offers malware blocking that uses the best information we can glean from all twenty threat intel providers, plus a whitelist of known-good major sites, to make sure that infected devices at your sites can't connect to C&C and start DDoSing people, and that credulous users won't be able to connect to phishing sites that will steal their information.

Google and Cloudflare do not do malware blocking. Cisco Umbrella/OpenDNS does.

What we recommend you do is to run a local caching resolver that performs QNAME minimization and DNS-over-TLS, provision it with plenty of cache, and only leak the minimum possible information out to a recursive resolver, and make sure that you agree with the privacy policy of the recursive resolver. Lots of folks use the combination of PiHole and Stubby for that purpose. One way you can tell whether people are monetizing your data is by seeing whether they recommend you connect your end-nodes directly to their service, or whether they recommend you put a caching resolver in front.

If you want to make this question more visible, you could post it to Quora, and I'll post the answer there as well.

Yoh Woody, how does Quad9 make money?

 

[unclebuk]

I thought part of using cloudflare was that they didn't collect any data like Google did...guess I was wrong!!!

According to Woody's website:

When you use Quad9 DNS Services, here is the full list of items that are included in our logs:

• Request domain name, e.g. example.net
• Record type of requested domain, e.g. A, AAAA, NS, MX, TXT, etc.
• Transport protocol on which the request arrived, i.e. TCP, UDP, and encryption status of the protocol
• Origin IP general geolocation information: i.e. geocode, region ID, city ID, and metro code
• Protocol version IP address – IPv4, or IPv6
• Response code sent, e.g. SUCCESS, SERVFAIL, NXDOMAIN, etc.
• Absolute arrival time
• Name of the Quad9-operated machine that processed this request
• Quad9 target IP to which this request was addressed (no relation to the user’s IP address)

We may keep the following data as summary information, including all the above EXCEPT for data about the DNS record requested:

• Currently-advertised BGP-summarized IP prefix/netmask of apparent client origin
• Autonomous system number (BGP ASN) of apparent client origin

All the above data may be kept in full or partial form in permanent archives

Also: (according to woody)
"There are exceptions to this storage model: In the event of events or observed behaviors which we deem malicious or anomalous, we may utilize more detailed logging to collect more specific IP address data in the process of normal network defense and mitigation. This collection and transmission off-site will be limited to IP addresses that we determine are involved in the event"

 

[coxhaus]

I don't want to pay for Cisco's Umbrella service for home use so QUAD9 has become my new standard for DNS.

 

[RMerlin]

inputed Norton DNS

The Norton DNS service was officially discontinued by Symantec last November.

 

[QuikSilver]

According to Woody's website:

When you use Quad9 DNS Services, here is the full list of items that are included in our logs:

• Request domain name, e.g. example.net
• Record type of requested domain, e.g. A, AAAA, NS, MX, TXT, etc.
• Transport protocol on which the request arrived, i.e. TCP, UDP, and encryption status of the protocol
• Origin IP general geolocation information: i.e. geocode, region ID, city ID, and metro code
• Protocol version IP address – IPv4, or IPv6
• Response code sent, e.g. SUCCESS, SERVFAIL, NXDOMAIN, etc.
• Absolute arrival time
• Name of the Quad9-operated machine that processed this request
• Quad9 target IP to which this request was addressed (no relation to the user’s IP address)

We may keep the following data as summary information, including all the above EXCEPT for data about the DNS record requested:

• Currently-advertised BGP-summarized IP prefix/netmask of apparent client origin
• Autonomous system number (BGP ASN) of apparent client origin

All the above data may be kept in full or partial form in permanent archives

Also: (according to woody)
"There are exceptions to this storage model: In the event of events or observed behaviors which we deem malicious or anomalous, we may utilize more detailed logging to collect more specific IP address data in the process of normal network defense and mitigation. This collection and transmission off-site will be limited to IP addresses that we determine are involved in the event"

@unclebuk Woody is part of Quad9 not Cloudlfare which is who @Kingp1n was referring to. Wanted to make that clear so it doesn't confuse anyone reading these forums. @Ryan K (Cloudflare) was speaking on behalf of Cloudflare.

 

[QuikSilver]

Yoh Woody, how does Quad9 make money?

@Bill Woodcock may help explain but Quad9 is considered a non-for profit company meaning the money they make goes back into the company to further their agenda. I'd imagine they get loans, or money from other companies for research purposes.

 

[dave14305]

I've set aside Quad9 with DoT for now because even with @EmeraldDeer's recommended 1900 ms timeout, I get more conn_shuts than I'd like with Quad9, and occassional DNS timeouts on clients, presumably due to Stubby reinitializing a TLS tunnel. A 2 second disconnect by Quad9 seems awfully quick compared to Cloudflare, for example.

Still using Quad9 over 53/udp, though.

 

[RMerlin]

I've set aside Quad9 with DoT for now because even with @EmeraldDeer's recommended 1900 ms timeout, I get more conn_shuts than I'd like with Quad9, and occassional DNS timeouts on clients, presumably due to Stubby reinitializing a TLS tunnel. A 2 second disconnect by Quad9 seems awfully quick compared to Cloudflare, for example.

Still using Quad9 over 53/udp, though.

Enable Round-Robin, it works better with these servers that have a short disconnect timer.

The Stubby authors are supposed to be working on a better way to handle this.

 

[dave14305]

Enable Round-Robin, it works better with these servers that have a short disconnect timer.

The Stubby authors are supposed to be working on a better way to handle this.

It was enabled, since this was on your firmware where it isn’t a choice. I only postconf’d the idle_timeout from 9000 to 1900.

Next time I’ll also enable dnsmasq logging to see what is really happening to the requests.

 

[Bill Woodcock]

Is a caching resolver the same as the 'Wan: Use local caching DNS server as system resolver (default: No)' in the Tools/Other Settings page in the Advanced Tweaks and Hacks section of RMerlin powered routers?

I have no idea, with respect to the specific router you're looking at. But it sounds like that configuration option is to have it run its own caching resolver, and use it, rather than sending every query out to an external recursive resolver. So, yes, I'd turn that feature on. If it starts giving you trouble, of course, RTFM and figure out what's going on, but as a first pass, I'd try turning on the local caching resolver and see how it goes.

That's good for the router itself. Then the question is how you get everything downstream from the router to also use it. In the DHCP server settings, I'd try making the downstream interface of the router the default DNS server, and see whether that works for your users. (I mean, I'd first test it by statically setting it as the DNS server for a downstream client and seeing whether that worked, then move on to making it the DHCP-provided one.)

What is QNAME minimization? How can this be implemented, if possible, on our routers today?

First, the QNAME, or Query Name, is the field in a DNS query that contains the string you're looking up. For instance, if you want to look up www.google.com, by default, your recursive resolver (assuming it has a completely cold cache) sends the QNAME "www.google.com" to a root nameserver; it gets a referral to Verisign's .com nameserver, and repeats the "www.google.com" query there; it gets a referral to Google's nameserver, and repeats the "www.google.com" nameserver there, finally getting A and AAAA records as answers. Which is why it's called a "recursive" nameserver... it recurses through authoritative nameservers walking down the hierarchy until it gets you the answer you want, and then it gives you back the answer, without bothering you with the in-between steps. And it caches everything along the way, so the next time you or someone else wants to look something up, it'll be much quicker.

The problem here is that there are tons and tons of people pcapping the packet stream in front of root and TLD nameservers, and recursive-to-authoritative traffic isn't really encrypted yet, except in extraordinary circumstances. So every one of those queries is immediately getting snarfed up by data-brokers, and sold to as many buyers as they can find. The search term you'd use to read about this particular flavor of bad behavior is "passive dns".

So, Stéphane Bortzmeyer wrote RFC 7816.

DNS Query Name Minimisation to Improve Privacy

This document describes a technique to improve DNS privacy, a
technique called "QNAME minimisation", where the DNS resolver no
longer sends the full original QNAME to the upstream name server.

https://tools.ietf.org/html/rfc7816

If a recursive nameserver is RFC 7816 compliant (and not many are yet), in the example above, it would only ask the root to resolve "com" and would only ask Verisign to resolve "google.com" saving the full "www.google.com" string for Google's own nameserver, which will expose the query to fewer taps.

So, the thing to do in the short term is to request RFC 7816 compliance from the router vendor, and if it's something you decide you care a lot about, build a separate caching (or full recursive) nameserver of your own, that does implement it.

Arrrrr. This thing wants posts of less than 10,000 characters, so I guess I'll split this up into multiple threaded replies.

 

[Bill Woodcock]

...and it also wants them to be more than fifteen seconds apart. Which makes sense, I guess.

Link encryption. I think this is DoT?

DNS-over-TLS (DoT) is the first of two IETF-standardized flavor of link encryption. DNScrypt is a non-standardized method that was implemented by OpenDNS quite a while ago, and still has a fair number of adherents. DNS-over-HTTPS (DoH) is the second standardized method.

From the most narrowly-construed technical perspective, all three solve the problem of encrypting DNS queries from you to a recursive resolver, and all three are supported on Quad9, because we are, as a matter of policy, agnostic at the technical level.

However, we recommend against use of DNScrypt because it was never formally standardized, so hasn't been through the degree of rigorous review needed to pass the security check phase of IETF standardization, and development has been frozen on it. So if new security issues were found with it now, they would likely not get fixed.

We also recommend against DNS-over-HTTPS, because while it's sufficient in theory, that wasn't what it was intended for in practice. HTTPS implementations are notoriously data-leaky, which means they're fingerprintable. So, people who are capturing and monetizing user data see DoH as serving their needs in a way DoT does not, because it allows them to track users as they move from IP address to IP address, because of fingerprintability of the HTTPS implementation. Also, the proponents of DoH are, principally CDNs, Content Delivery Networks, who see consolidation of DNS service onto their "platform" as a tool for violating net neutrality. If you were CDN A, and you could convince people to send you their DNS queries (particularly over DoH), you would have special knowledge of what queries were coming next in any sequence of web page loads, and you could pre-feed those answers down to the client as "additional information," yielding faster performance for clients loading only that content which was hosted on your platform, and slower performance for all the content hosted on your competitors' platforms, even before you intentionally slowed down queries yielding answers pointing at your competitors. Moreover, any time there was a choice between directing clients to results on your platform and results on your competitors' platforms, you could always direct them to your own, increasing your revenue at the client's (and content publisher's) expense.

So, between a protocol which is privacy-enhancing (DoT) and one which is, by intent, privacy-violating (DoH), we of course recommend DoT, though users are free to use any of the three to talk to us.

How important is it to 'compartmentalize DNS queries' and, do our routers do that for our devices now?

I would say that, from a privacy perspective, it's very important. Nearly all recursive resolvers, many authoritative nameservers, and a lot of people sitting on the wire looking at traffic crossing the internet are deanonymizing and monetizing DNS queries. The stream of DNS queries coming out of your machines reveal not just what web pages you're looking at, but what advertisements you're being shown (which in turn reveals what other people already know about you), and every piece of software on your machine that's checking for updates, whether that's something legitimate like Photoshop, or a piece of malware that's looking for C&C.

The first and most important thing you can do is to put a caching resolver in front of your users, so they're all hitting the same cache, subsequent queries for the same domain aren't revealed, and their IP addresses are masked by that of the cache. Also, of course, they receive much better performance that way, and they're resilient against poor performance or reliability on the Internet connection. Make sure the cache fits in RAM, so it doesn't swap to disk and destroy performance, and make sure the cache is big enough that you're not getting discards before TTL expiries on queries that wind up getting repeated. This is all performance-tuning that's easy to do on a stand-alone nameserver, but may not be available to you at that level of granularity on whatever's bundled in your router. At this point, PiHole is the caching resolver package I'd recommend most for home and small office use. It gives you all of those capabilities, and is well supported, and gets extended with new features all the time. It also does a very good job of ad-blocking.

QNAME minimization is an excellent next step, but it's one of those features that will be integrated into all privacy-aware nameserver implementations shortly, so it may be easiest to just wait for it to show up in whatever software you're otherwise using, and check the config box at that point, rather than going looking for something specific now.

And then, of course, you need to weigh the relative merits of sending your remaining queries to a recursive resolver (like Quad9, or Cisco, or Cloudflare, or Google), versus doing your own recursion, and sending QNAME-minimized queries directly to authoritative servers. If you use a recursor, it will give you an additional layer (pretty complete) of anonymity in any traffic that makes it through that cache to the authoritatives. It'll also give you a performance boost. But it'll route all of your traffic through one place (as a VPN would) so you need to really trust whoever's operating it to have your best interests at heart. On the other hand, if you do your own recursion and send your queries to authoritatives, those will all have to be in-the-clear, and they'll all have your nameserver's IP address on them. So you don't create any point of concentration, but you wind up exposing pretty much everything in plain-text, with your own IP address on it. So, we recommend using an external recursor, but being absolutely sure you understand their privacy policies and practices if you do so.

If you were going to get more baroque, you could look at distributing your queries across different nameservers in NS sets, using Tor or other VPNs to carry your traffic, etc. But in all likelihood, none of you have any specific need to carry things that far.

I think all of the above points to the goal of post 67 which states exactly as I hope I'm operating my network; "Minimize the amount you have to trust anyone else.".

Yes, exactly. Our goal as a not-for-profit is to make sure that everyone's privacy is protected as much as possible. That means that it's our goal to see as little of your traffic as possible. We want you to cache everything you can, and extend as little trust as possible, to us, or to anyone else. We want you to be very skeptical of claims, since so many of them have turned out to be false (cf. Snowden revelations regarding Google, for instance). We're in the process of reincorporating from the US to Switzerland, so that our privacy regulation compliance will no longer be voluntary, but will be guaranteed by a strong national privacy regulator. Because assertions that aren't backed by strong legal enforcement are really just assertions.

Oh, one thing I hadn't mentioned before, that's a big differentiator between recursive resolver systems: Quad9 and Verisign's Public DNS are deployed back-to-back with huge authoritative DNS clusters, serving two root nameservers and hundreds of millions of zones and (in Quad9's case) many hundreds of TLDs. The others aren't. Again many people default to considering this from a performance perspective, because it removes the latency between the recursive and authoritative servers; but what's really important is that it also collapses the attack surface. If there's no space between them, there's no space for an attacker to operate, nothing for them to attack. Roughly, here's what that looks like:

To be clear, PCH and Verisign aren't extending any special preference to each other that we wouldn't to other parties; to some degree Cisco and Google also interconnect with us and Verisign relatively closely, but they don't have any significant authoritative content of their own.

So I'd say that's another huge difference between different recursive resolvers.

 

[Bill Woodcock]

Yoh Woody, how does Quad9 make money?

It doesn't. It's a public-benefit not-for-profit foundation, which exists for the sole purpose of doing what it's doing, which is providing a recursive resolver that's GDPR compliant by dint of not collecting any personal information.

Like most non-profits, it's supported by donations. The largest donors are PCH, the non-profit that I run as my day-job, NTT, which is the Japanese national long-distance phone company, IBM, and SWITCH, the Swiss national research & education network. Donors do not receive anything that the public does not also receive.

 

[Bill Woodcock]

According to Woody's website:
When you use Quad9 DNS Services, here is the full list of items that are included in our logs:

• Request domain name, e.g. example.net
• Record type of requested domain, e.g. A, AAAA, NS, MX, TXT, etc.
• Transport protocol on which the request arrived, i.e. TCP, UDP, and encryption status of the protocol...

As I said before, the web site is a cobbler's child, and that privacy policy was written by lawyers, who are used to butt-covering, rather than engineers, who tend more toward the minimal correct solution, so it's sorely in need of a paring-down edit. The main thing I'd point out about it is that they used the word "logs" where I'd have used the word "counters." There's far, far, far too much going by to log, in the sense of keeping a set set of associated records for each query. So in the case of example.net above, we have a counter of how many queries there have been for example.net within a five-minute window. Separately, we have a counter of how many queries have been answered with an A record within a five-minute window. Separately from that, how many queries have been answered with a AAAA record within a five-minute window.

By analogy, if we were counting cars going down the street, we'd have a bunch of separate counters: one of how many Fords there were, one of how many blue cars there were, and one of how many pickups there were, but we wouldn't be able to discern from that how many blue Ford pickups there were, unless the value of each counter was exactly 1, and we knew the total to also be 1. But that doesn't happen. These are counters in the hundreds of millions.

We think we're only counting things that are useful, but if anyone has an argument that any of those aren't useful for us to be counting, we're also perfectly happy to consider dropping it from the list and purging any stored records. There's a certain amount that's helpful to us in understanding and countering attacks against the system, but speaking generally, collecting and storing any sort of data at this scale is expensive, and we only get paid for not storing data, not for storing data.

 

[Bill Woodcock]

I ... did a few checks on www.dnsleaktest.com and both test results show the DNS is resolved by Level 3 Communications and NTT Singapore Pte.

What you're seeing is that our recursive server which handled your queries is using Level3 (actually CenturyLink, post-acquisition) and NTT for transit. Since it's NTT Singapore, it's very likely that your query is being handled by our Singapore node, which means that your transit provider is choosing to haul it there, rather than to our Bangkok or Kuala Lumpur nodes, which are closer.

Since we choose our outbound path (which is what's being recorded here) based on what routes we have to the destination, we'd only be choosing to use transit if the transit provider of the instrumented authoritative server used for your measurement wasn't among our Singapore peers. Indeed, a traceroute shows it to be single-homed behind Cogent in Atlanta. So from Singapore, reaching a single instance of something in Atlanta via Level3 or NTT is actually a pretty efficient way to get there.

If you were trying to resolve a regular domain name, rather than trying to get a packet to Atlanta, it's much more likely that your query would stay within-region, since there'd more likely be an instance of the authoritative in-region.

 

[Bill Woodcock]

When I run DNSLeakTest using the above, I see exactly what I expect.

View attachment 18139

...and the reason why this shows WoodyNet instead of Level3 and NTT is that we've got a direct interconnect with Cogent in Miami where we handed it off. So I'm guessing you're either somewhere reasonably close to Miami, or somewhere in the Caribbean or Latin America, getting transit from someone who only peers in Miami.

 

[L&LD]

@Bill Woodcock, thank you for answering the questions I had so completely.

I will be studying these for a while.

 

[RMerlin]

So, the thing to do in the short term is to request RFC 7816 compliance from the router vendor,

The majority of routers that have any built-in caching resolver use dnsmasq, which doesn't do recursive lookups, it mostly acts as a caching forwarder. So the burden of RFC7816 lies in the upstream servers used by the forwarder.

This is the same case AFAIK with Stubby, which is probably the most popular stub resolver that supports DoT.

 

[RMerlin]

@Bill Woodcock , is there any benefit in setting both 9.9.9.9 and 149.112.112.112 , or are they routed to the exact same node, and the second address is only intended for clients that absolutely require two different nameservers in their configuration?

 

[Citizen93d]

(new account as this is the first time I participate here, found this thread via RMerlin's tweet)

Quad9 is an interesting service. I personally don't use because it's backed by some entities like the City of London Police (not to be confused with the Metropolitan Police), known for their overreach online.

Quad9 was created, in part, by the Global Cyber Alliance (GCA), a non-profit that was founded by Manhattan District Attorney Cy Vance, Jr., the City of London Police, and the Center for Internet Security, with a seed investment of asset forfeiture funds provided by the Manhattan District Attorney. (source)

While I understand that running a public DNS resolver costs money and assume Mr Bill has good intentions, Quad9 is associated with entities that sometimes abuse their power or are aligned with businesses interests (City of London Police for example, a police force located in the business-controlled City of London) and for that reason I wont use it.

Just to be clear, I'm not saying that Quad9 share data with these entities. It's the association with them that makes me avoid the service.

 

[skeal]

I don't blindly do anything, nor do I follow the leader, but the information in this thread has converted me over to Quad9. They clearly have less need for information than Cloudflare. The experimental IP range use bothers me as well. I'm in full testing mode as we speak.