From the logging I was seeing SmartThings performing queries, but never getting a response. These were going to the router, which is advertised as the DNS server, so it's not clear why the response was not being received unless it was ignored. My client could lookup the same entry and resolve.I don’t think that’s the correct conclusion to draw although I think both symptoms you’re observing are both due to the recently enabled strict checking of dnssec.
Your SmartThings hub is not working simply because the DNs lookups failed (because it’s querying a custom DNS server instead of using the router supplied one?) You can look further into this.
As for dnscrypt-proxy, it probably fails to start because it is using doing some initial checks and perhaps the change in dnsmasq broke it, but that’s the direction I’d go for further troubleshooting.
I’m not sure what you mean by “noted in the release notes”, these are alpha versions of the firmware exactly for the purpose of working out kinks like this. Ideally it should be resolved before release and noted only if there’s is some limitation stopping it from being fixed.
It’ll be cool if you can look further into your SmartHub and also figure out why dnscrypt-proxy is failing to start (more verbose logging maybe)?
I'm aware that the alpha version is for finding these kinks. If the DNSSEC changes will persist, it is at that time that it would be noted, otherwise reverting to non-enforced (or opportunistic) DNSSEC would be the recommended approach.