BuddyNS

About reachability

Some users recently inquired about our rules for e-mail addresses. In this post we shed some light about this.

The background

BuddyNS is a DNS reliability service. Because issues in an organization’s DNS affect all organization’s services in a waterfall effect, we take smooth running of your domains extremely seriously.

To this end, our systems perform routine checks on your domain’s health and report to you as soon as something goes wrong. Our experience over the years showed that e-mails addresses entered by users are or become unreachable with ridiculous frequency, because of nasty spam filters, disposable e-mails, unattended mailboxes, quotas, broken aliases, or a gazillion other reasons.

If we report you a problem and you cannot take action on it, your domain names suffer downtime, and the situation ends up in stress and disappointment on your side. Even though this is up to the user’s poor choice of his e-mail contact, we refuse to provide a service in such miserable conditions.

So we first introduced the free SMS notification service. A large fraction of users choose to not register their mobile phone, and we respect this privacy choice. Therefore, BuddyNS now assesses the reliability of an e-mail address and permits activation of one zone with our system only if this passes some criterias.

If you are unwilling to provide a reliable e-mail address, one compliant with these criterias, look further for your secondary DNS provider.

Good e-mail addresses

When you activate a new zone on BuddyNS, either from the zone activation page or from inside the BuddyBoard, BuddyNS may return you one of the following errors:

• You’ll get critical notifications here, please use a better contact!

  The e-mail address you provided for your account blatantly looks disposable, e.g. “john-spam@myhost.com” or “john-buddyns@gmail.com”. Our experience shows you won’t frequently check this mailbox, or you might at some point forget it or discontinue it.

• Contacts *@myzone.com unreliable for hosting myzone.com!

  You have provided john@myzone.com as account contact, but want to activate myzone.com . If any critical DNS problem occurs with myzone.com, email for it will be unreachable and you won’t receive any notifications.

Older accounts

Notice that, if you registered your account before mid April 2013, your e-mail contact might already be affected by these risks. In this case, next time you attempt to activate a zone, BuddyNS will issue these errors.

If that’s your case, head over to the BuddyBoard, Account pane, and fix your e-mail address. Then attempt activating your new zone again.

Help us help you: be reachable

BuddyNS won’t send you e-mails that are not in your direct interest, so you may not read from us for extended periods of time. It’s ultimately your responsibility to make sure that the e-mail contact you gave us for running your zones is reliable.

Always keep your BuddyNS e-mail address up-to-date. If you have one, we advise you to register your mobile number in the BuddyBoard as well. We send the most critical DNS notifications (such as “you ran out of traffic, all zones blocked!”) there. This is a free service.

Needless to say, we never share our users’ contacts with any third party.

Koalas eat maple leaves in the BuddyNS forest

We recently augmented the BuddyNS DNS network with two new nodes:

• g.ns.buddyns.com in Vancouver, Canada
• h.ns.buddyns.com in Adelaide, Australia

Here they are on a map: red are the new additions:

Which servers should I use now?

BuddyNS users look for reliability of their domain names. For best reliability, delegate your zones to as many BuddyNS servers as possible.

Seven nodes exceeds the capacity of most registries, so you’ll have to pick which servers to delegate by hand. Delegating to three is a good starting point.

Here’s some ideas for how to pick these.

Best speed

Figure out where your users mostly come from geographically, and pick the BuddyNS servers closest to that. For example, if most of your visitors are European, pick server c.ns and e.ns for Europe, and b.ns for America.

Fairest distribution

Pick one BuddyNS server per continent, for example f.ns for Asia and Oceania, c.ns for Europe, and b.ns for America.

Easiest setup

Pick three random BuddyNS servers. This works just as well :)

How about GeoStats ?

The addition of these new servers does not affect you if you are having BuddyNS monitor DNS traffic for some of your zones.

The BuddyNS cluster is already set up to gather data from every publisher and aggregate it into your graphs. Traffic hitting the new nodes will simply be part of it.

How about Vanity DNS ?

If you have Vanity DNS set up for some of your zones and you desire to leverage some of the new nodes, e.g. for better proximity, then all you need to do is update the address of your mapping to the new server.

For example, if your current setup delegates your zone to:

• ns1.myorganization.com = 12.13.14.15 # your own server
• ns2.myorganization.com = 209.177.145.51 # BuddyNS server d.ns (New York City)
• ns3.myorganization.com = 203.142.25.114 # BuddyNS server f.ns (Singapore)

and you want to replace the Singapore node with the Australian node, just replace the address for ns3.myorganization.com from 203.142.25.114 to 119.252.20.56 (h.ns in Australia) in the zone file you keep for myorganization.com.

See Vanity DNS setup for more infos.

What about IPv6 ?

All new servers declare and respond over native IPv6 interfaces. Enjoy!

Goodbye PDQ updates!

The problem BuddyNS started out to resolve was the inherent inconsistency periods dragged in by Secondary DNS. For over 3 decades, administrators willing to employ a Secondary DNS service were afflicted by hour- or even day-long periods, after any DNS change, before the change became available to all users. When the update regarded new services, or worse a migration of a failed server, this delay would burn the admin like chilipowder in the eye.

PDQ updates were the first solution BuddyNS implemented for this problem. With an e-mail to a magic address, the DNS user could request immediate updates of his zones. This could be done simply, portably, and without need for more complex machinery like NOTIFY on the user side.

PDQ served BuddyNS for years, accompanied SyncNOW! once we introduced it, and are End of life today.

Why are we discontinuing PDQs? Since we introduced SyncNOW!, people’s custom to the Web made most users switch progressively to it. PDQ serves today few requests per day; it has become a surpassed technology are we are pruning it out to make room for the new.

What does this mean to you? BuddyNS maintains its immediate synchronization abilities with NOTIFY and SyncNOW. If you subscribed onto BuddyNS in the last 2 years, this probably makes no difference to you already. If you subscribed before, and you still use PDQ to update your zones, do switch to SyncNOW: log into your BuddyBoard, follow the “Zones” pane, and click the SyncNOW button on the top-right.

Thanks PDQ, goodbye!

Updates to your BuddyBoard

Part of the BuddyNS team met last week in Tarifa, Spain, to have one week where we alternated coding sprints and kite surfing.

Luckily for all of our users, the weather was not friendly for a few days, so more news made it to life than expected :)

First, we resolved an issue affecting a few GeoStats users where graphs would occasionally go mute, or display significantly less traffic than real. Sorry folks! This was up to a scalability issue in the logic for generating GeoStats configuration data to distribute to each DNS publisher. Michele optimized it away and you can rely on your graphs again for profiling your traffic patterns and investigating unpredicted traffic spikes.

Second, we performed some improvements to your User Experience on the BuddyBoard (BuddyNS’ user control panel). All of this is live in your account already, and here’s a quick walkthrough to familiarize you with these changes.

There are 3 main newcomers in your BuddyBoard:

#1 – Consistent BuddyBoard alerts

All alerts displayed by the BuddyBoard are redesigned and collected under a single, consistent framework. They look better and behave better.

The BuddyBoard used to display some messages about events relevant to your account. They could be alerts – such as “Your traffic quota is depleted!” – or informational – such as our “Hey, a donation to BuddyNS is less than tomorrow’s coffee!”. They used to have custom styles depending on the event type or the place where they’d be displayed.

They are now uniform, intuitive and consistent. Here’s their semantics:

• alerts on blue background are informational
• alerts on yellow background are warnings – conditions you should look into
• alerts on red background are errors – conditions you should take action upon ASAP

#2 – Smart BuddyBoard alerts

We are progressively adding new informational alerts. The new “alerts” framework makes room for smarter event-detection logic. For example, the BuddyBoard will now warn you weeks in advance that your traffic quota is likely to run out.

Smarter events give you more insight and more control over what’s happening with your account and zones. Here’s some of the new alerts you may see:

• recent updates on your zones
• recently arisen errors on your zones
• recently solved errors
• your traffic quota will deplete before the end of the month (forecast)

Whenever you feel uncertain how to respond to one such event, feel free to contact us for help at support@buddyns.com .

#3 – The delegation bar

A new progress bar, which we call Delegation bar, now shows up in your BuddyBoard “Zones” pane, that loads green or red when the page is loaded. This progress bar shows what’s going on with the loading of the page.

The “Zones” page gives you a real-time configuration status of your zones. The “Zone declares BuddyNS” and “Registry declares BuddyNS” fields tell you if you configured delegation correctly.

Checking delegation for one zone is a many-step process where several timeouts may occur, so the BuddyBoard performs asynchronously after the page is loaded. Zones initially load gray, and then get colored blue (OK) or red (ERROR) when delegation information is available.

Two things about this were confusing to some users: why zones would change color, and why some of them took many seconds before doing so.

The Delegation bar gives an immediate, visual answer for both: you see what’s going on, what it is about and what is the timing and progress for it. As an extra, the Delegation bar gives you a quick prospect of what fraction of your zones need delegation adjustments.

Enjoy! The BuddyNS team

Anonymous asked: How to get started with BuddyNS? I have a linux box running Bind9 with 10 domains in it. How to setup the AXFR or SyncNow I saw you discussing before? Thanks!

You receive these instructions via e-mail after you activate a zone on http://www.buddyns.com/activation/ .

To activate zone transfer (AXFR), see http://www.buddyns.com/support/setup/#enable-transfers and click on “I use BIND on my primary DNS”.

SyncNOW! does not need setup. Once your zones are on BuddyNS, you click the SyncNOW! button in your BuddyBoard and BuddyNS will immediately synchronize all zones in your account from your indicated master.

-t

Anonymous asked: Hi, I just had a faithful secondary nameserver fail today and I'm looking into your services. I have 3 PTR records that are used for my 3 email servers. Does your service handle that and if so how would I get that to work?Thanks,Jay

You can serve PTR zones with BuddyNS: just register them in their canonical name. Eg, for 1.2.3.4, register 4.3.2.1.in-addr.arpa. Make sure you get delegation right for your PTRs, the Delegation Lab is your friend.

cheers

-t

Say Aloha to the BuddyNS Delegation Lab!

Today we introduce the BuddyNS Delegation Lab.

The BuddyNS Delegation Lab makes your DNS life simpler by helping you find hard-to-locate delegation problems easily. It is a public service accessible to everyone – BuddyNS user or not – and for every zone.

The full DNS delegation chain for a zone usually involves over 20 servers and 3-6 recursion hops. The BuddyNS delegation Lab goes through this for you, performs checks at every step, and gives you the result in a nice, easy-to-read graphical chart.

Here’s some examples of what you can find with the Delegation Lab:

• Dead or misconfigured servers in your delegation chain
• Inconsistencies in your nameserver data between registry and authority
• Inconsistencies in the zone versions served by your nameservers
• Lack of correct delegation to BuddyNS

The rule of thumb is: if you see colors other than blue, green and red you’re called for action.

Sane delegation

Let’s look at an example. We want to check zone buddyns.com, and there’s 3 alternative ways to do that:

• we visit http://www.buddyns.com/delegation-lab/ and enter buddyns.com
• (or) we type buddyns.com into the URL directly: http://www.buddyns.com/delegation-lab/buddyns.com
• (or) we reach the BuddyBoard zones page and find the link ready (as two little test tubes) for each of our registered zones:

BuddyNS will take a few seconds to explore the DNS delegation tree for buddyns.com, then it will display this:

This graph shows the DNS traversal to resolve buddyns.com, i.e. which nameservers have been inquired about buddyns.com starting from the root zone (“.”) down to the authoritative nameserver for it.

Each black arrow indicates which server BuddyNS chose to inquire next, so the most interesting part is “the tail” of the graph, because that’s your area of competence. This graph shows:

• Red balls indicate a zone
• Blue boxes attached to zones indicate nameservers serving the zone
• Green boxes indicate nameservers that are running on BuddyNS infrastructure
• the Gray node Authority is the final authoritative answer given to clients about what DNS servers serve the zone.

So the least information this graph gives us is:

• clients have a way through the DNS hierarchy to reach buddyns.com
• the list of nameservers for buddyns.com announced by registry is consistent with the list announced by Authority
• the zone makes use of BuddyNS for DNS replication

This looks like there’s nothing calling for attention, so let’s look at another example.

Insane delegation

This is an example from a real-world zone particularly gifted with issues:

Here’s some new colors, and they carry significant information. Let’s see what they mean:

• Dark gray nodes on zone node: these two servers were queried about the zone, but they failed to respond. Possible scenarios:

  • they are dead
  • they have networking problems (firewall? routing?) making them unreachable from the Internet or unable to send back responses
  • they are delegated for the zone, but do not actually serve it
  • they are overloaded and replied too long after the Delegation Lab timed them out.

  What to do: Check on each unresponsive server. If you can’t fix it, remove it from the list of nameservers for zone to avoid undesired delays and timeouts to users.

• Yellow nodes : the authority and the registry declare inconsistent nameservers; these two (yellow) servers are known only to the authority. Possible scenarios:

  • new nameservers have been added in the zone, but the registry has not been updated. This may mean that new available DNS servers are not announced to clients.
  • discontinued DNS servers have been removed from the registry (or zone). This case is particularly dangerous: if the discontinued server IP is now owned by another organization, this organization can control your services.
  • servers were supposed to match, but a typo made it into one side. Wrong names can incur delays and timeouts at users, delegation to untrusted parties, and break security if the nameserver operates DNSCurve encryption.

  What to do: Merge nameservers announced by the registry and the zone. Determine which ones must actually serve the zone and remove everything else to end up with consistent delegation.

• Purple nodes highlight nameservers that are publishing an outdated version of the zone. Possible scenarios:

  • records for new services have been introduced: clients that reach the up-to-date nameserver(s) will successfully reach the new services, while clients that reach the outdated nameserver(s) will receive an error. This causes confusing unpredictable behaviors for clients; also, clients served by the outdated server will fail to resolve the new records after you fix consistency, and until the NXDOMAIN TTL for the zone expires (often several hours).
  • records for existing services have been changed, or removed: clients that reach the outdated nameserver(s) will resolve to wrong addresses. If these addresses are no longer in control of your organization, some third party could deface or steal sensitive data to your organization or your users.

  What to do: check on why the outdated server(s) are not in sync with the master. If you cannot fix the problem, remove them from the list. If they are third-party service with lazy synchronization, switch to BuddyNS.

Sane or insane?

The BuddyNS Delegation Lab enables you to verify correct DNS delegation of any zone in the blink of an eye. If you have any delegation issue, the Delegation Lab allows you to promptly locate it and give depth to it.

Comments? Tweet or write in to [support@buddyns.com](mailto:support@buddyns.com?subject=[blog] delegation lab) !

Sluggish AXFR for some users

Some users reported slow synchronization in the last two days. They were updating their zone(s) and expecting BuddyNS to synchronize within 10 minutes. Or they were pushing the SyncNOW! button and expecting BuddyNS to synchronize within 2 minutes. But BuddyNS wouldn’t.

Instead, BuddyNS was taking tens of minutes or even some hours to synchronize. This is exactly how day-to-day life is with every traditional DNS service; but since BuddyNS users have a higher standard, some got loud on this matter :)

So what happened there? The short story is, we fixed this issue some hours ago and you can bring your DNS expectations back up to the usual standards.

The long story has some interesting background on BuddyNS’ problem solving.

The triggering fact for this sluggish AXFR issue was the software at a user with a large number of zones issuing continuous SyncNOW! requests. Our backend is designed to handle such cases by using these techniques:

• a zone is synced at most once per minute regardless of how many SyncNOW! requests occurred for it.
• when syncs are performed, their rate (transaction/second) is limited to avoid overloads on any user’s master.
• zones to sync are picked with a random policy at each step for reliability matters.

This design is what guaranteed slick and prompt updates ever so far.

In this case, however, the continuous rescheduling of so many zones triggered a quasi-starvation condition: zones to sync for other users were getting “drown” in the big mix, having a tiny probability of being picked, and therefore pending for sync for a long time.

Starvation is a well-known problem, but not always blatant. In case of AXFRs, the probability for each zone to undergo sync at any time is expected very low, so that even with a big set of zones, the pool of new zones to update every minute is petite. For example, if 50 thousand zones get updated twice a month, then every minute averages to only 2.3 zones to update (# zones / # minutes per 2 weeks).

If the math says 2, then engineer for 100, and keep an eye the what-if 500 case. But in this exceptional event every minute had several hundred zones re-scheduled over and over again.

At BuddyNS, whenever we face an exceptional case problem, we ask ourselves:

Is this likely enough to be addressed? Can we ignore it (KISS)?

Addressing a problem comes with drawbacks: complicating something that works is always a Bad Thing®, and every new LoC has a probability of carrying new bugs attached.

Yet, most often there is a way in between: most hard problems have a trivial solution if you

• accept to not solve the problem entirely, just “most” of it
• make the effort to look at it from non-traditional perspectives

In this case, we just decided to use a firewall policy to rate-limit the number of SyncNOW! (and NOTIFY) requests for each party over a definite amount of time. SyncNOW! and NOTIFY remain fully functional for normal use, and unfairness can occur for a few minutes if a client’s software goes crazy. But it won’t last longer, and no code needs to be written.

Finding one such solution can take an hour of brainstorming, but save days of work, testing, and monitoring.

We found that accepting pragmatic solutions greatly simplifies life and maintains a system clean and maintainable. Finding Egg of Columbus solutions is the harder part, along with accepting the part of the problem which will stay unsolved, but comfort grows with both the more you do this.

Anonymous asked: I have "connection timed out; no servers could be reached" with your DNS servers. What is wrong?

Hi Anonymous,

This means that your zone is not served by BuddyNS. Possible reasons:

• it is not activated on BuddyNS (see activation)
• your master does not provide zone data to BuddyNS (see AXFR setup)
• you depleted your traffic quota and new queries are not answered (see traffic quotas)

Refer to the BuddyBoard to know which reason applies to your case.

-m

D-Day May 5, 2012

Takes place on May 5, 2012 at 10.00 to 13.00 UTC. It addresses:

• Scalability
• User eXperience
• IPv6

This is a major deployment and we are thrilled :)

During deployment these services may be not available:

• Registration of new zones
• Removal of existing zones
• SyncNOW
• Traffic statistics

Nota bene: your zones will keep running on our DNS servers like a charm.

BuddyNS traffic quotas: how it all works

Starting with May 1, 2012 (May Day! :) ), BuddyNS enforces the traffic quotas advertised throughout the website.

This has some important consequences, so read through!

First, what’s visually new about this: Two cool big gauges in the BuddyBoard show you your zone and traffic consumption in the blink of an eye:

Second, in case you don’t occasionally access the BuddyBoard, BuddyNS makes sure to send you advices on your traffic progress:

1. when 60% of your traffic quota is used up (e-mail)
2. when 90% of your traffic quota is used up (e-mail)
3. when 100% of your quota is used up (e-mail + SMS)

When your quota is entirely used up, BuddyNS stops answering queries for your zones.

Having BuddyNS stop serving your domains has delicate consequences:

• if you run a “stealth primary” (your NS records ONLY refer BuddyNS), all your services become suddenly globally unreachable
• if your primary runs along with BuddyNS, your primary gets all DNS load at once, and all your services become suddenly unreachable if any hiccup occurs on it
• in either case, your clients experience delays or failures when accessing your services

So, what should you do to take care of your traffic consumption? Some obvious advices:

• get aware of what is your average traffic. The BuddyBoard makes this trivial for you: just throw a few glances at it over the first weeks.
• pay attention to e-mail notifications from BuddyNS! Mark “support@buddyns.com” in big bold red in your e-mail program if necessary; BuddyNS e-mails your for important stuff only.
• register your mobile number for safety. We see e-mail notifications nullified by spam filters, mailbox quotas, and discontinued mailboxes every day, and can’t do anything about it.

If your traffic is above quota as a one-time case, e.g. for a publicity spree at your website, perform a small donation and drop us a line – we can provisionally unlock your quota.

If your account’s traffic is consistently above the quota, you can upgrade your quota with the “Subscribe” button on the BuddyBoard. Alternatively, you can reduce the traffic you incur by removing some zones from your BuddyNS account, or removing some BuddyNS servers from your zones’ NS records, at some reliability expense.

Finally, why did we introduce traffic control only now?

When we introduced traffic quotas back in 2011, we decided to leave it up to users to pick their account type. Implementing an accurate traffic accounting system was not only a challenge on the technical side, for a high-volume distributed infrastructure, but also a challenge on the User eXperience side, for providing enough control and notices to ensure safety yet without annoying users. So we trusted users to be familiar with their traffic and to switch to the most appropriate account class accordingly.

Many e-mails to support later, we realized that most users are actually clueless about their traffic, and tried to lay down a design to tackle this. The new traffic accounting system is the completion of this; it introduces precise control over the infrastructure’s usage, fine feedback to each user, and ultimately guarantees fairness to those users (bravo!) who did track their traffic and picked their account class spontaneously.

Enjoy!

The BuddyNS team
Follow the status of BuddyNS @ http://twitter.com/BuddyNS