DNS Name Failover for Highly Available AWS Services

DNS Name Failover for Highly Available AWS Services

Highly available AWS services are supposed to be build covering multiple Availability Zones (AZ). Highly available AWS services with single point of failures (example: traditional, non parallel database) need to implement their single point of failure in each Availability and assure that the state of the services stays syncronized.

The second task is to assure that the backup services becomes available to end consumers in the case of a failure of the primary service.

One way to solve this problem is to reassign a given IP adress to the standby server. AWS currently offers two ways to accomplish this failover. The IP Failover scenarios are described here.

This document focuses on a name based failover in a private VPC with access to an on premises intranet.

Targeted architecures for Route53 Failover

This document describes how to switch the IP address for a given name entry in the AWS specific Domain Naming service Route53. This approach will work for the following architectures:

  • Consumers of a highly available service connect through the services through name exclusively.
  • Consumers of such a service have a comain name service configured which uses the AWS Route52 information
    • They use a public AWS IP address with a publicly registered domain
    • They have an on premises intranet which uses AWS Route53 as a secondary domain name server.
  • End consumers are able to reconnect a network session in a case of failure

The solution presented on the following pages will resonate more with DNS users. A similar solution using Active Directory and the Actice Directory Connector is being presented here.

Stefan Schneider Fri, 08/12/2016 - 12:35

Creation of a Route53 Service in a private VPC

Creation of a Route53 Service in a private VPC

Enable your VPC to work with Route53. Skip this step if you already use Route53.

The AWS Command Line Interface (CLI) allows to list all VPCs with the following command:

aws ec2 describe-vpcs
Enable DNS Resolution.

The following commands will make the instances in your VPC use the domain name entries from Route53. This is most likely required since some of the consumers may be part of the VPC.

Use the AWS console. Move to the VPC screen. Make a right mouse click above the VPC which you plan to enable. Select "Edit DNS Resolution"

Edit DNS Resolutions It will then show a modal dialog in which you will have to click on "Yes" and "Save"

The AWS Command Line Interface (CLI) allows to perform this operation though the following command:

aws ec2 modify-vpc-attribute --vpc-id <value> --enable-dns-support
 Edit DNS Hostnames The next step happens as well through the AWS VPC console. Hover the mouse above your VPC. Make a mouse right click. Select ""
Edit DNS Hostnames It will show a modal dialog. Select "Yes" and Save your data entry.

 

The AWS CLI allows to perform this operation though the following command:

aws ec2 modify-vpc-attribute --vpc-id <value> --enable-dns-hostnames

Important: Note down the name of your VPC or tag it. The name will be needed later on.

Creation of the Route53 Hosted Zone (The Domain Name Server)

Use the AWS console. Move to the Route53 screen. Click on the Create Hosted Zone button. You will see a dialog like the following one:

 create hosted zone

We assume that you will want to use this domain name server for intranet case only. Fill the fields with the following values:

  • Domain Name: Consider to pick a subdomain from your intranet domain. Your intranet domain name server will have to use this server for name resolution for your VPC only
  • Comment: A comment :-)
  • Type: Select Private Hosted Zone for Amazon VPC
  • VPC ID: Enter the VPC-ID from the VPC you plan to associate the hosted zone with

Save everything through using the Create button.

Stefan Schneider Fri, 08/19/2016 - 14:49

Changing the IP Address for a Host Name Entry in Route53

Changing the IP Address for a Host Name Entry in Route53

The failover can be execute by changing the IP address for a given host name. This means a A record has to be introducted or modified.

This can be done through a CLI command. The information required is:

  • Route53 hosted zone id: for example HOSTED-ZONE-ID
  • name of entry: for example myservice
  • IP address for entry: for example 10.79.252.13 
  • domain name: for example myvpc.mycompany.com
  • Time to life (TTL): for example 10s 

Prework

Find your hosted zone through using the AWS console or the AWS CLI command:

aws route53 list-hosted-zones

Create a file like change-resource-record-sets.json:

{
"Comment": "Update record to reflect new IP address for a system ",
"Changes": [
{
"Action": "UPSERT",
"ResourceRecordSet": {
"Name": "myservice.myvpc.mycompany.com",
"Type": "A",
"TTL": 10,
"ResourceRecords": [
{
"Value": "10.79.252.13"
}
]
}
}
]
}

Replace the string myservice.myvpc.mycompany.com with the host name and the domain which matches your requirements.

Replace 10.79.252.13 with the IP address where your service is being provided.

Changing the A record in Route53

Use the following AWS CLI command to implement the change:

aws route53 change-resource-record-sets --hosted-zone-id HOSTED-ZONE-ID --change-batch file:///mypath/change-resource-record-sets.json

Replace HOSTED-ZONE-ID with the identifier for your hosted zone. Pick the correct path to your json configuration file.

This command can be used to create an A record initially. It will work as well if the record already exists. It will then update the IP address and the TTL value.

The command will return a transaction Id which should be caught...

Checking Progess of the Update

The Route53 update takes as while (~20s). The successfull completion of the transaction can be checked with the following command:

aws route53 get-change --id <value>

The result will provide a status field which is PENDING or INSYNC.

 

Stefan Schneider Fri, 08/19/2016 - 14:55

Propagating the Name Change back to the On-Premises Intranet

Propagating the Name Change back to the On-Premises Intranet

We assume here that an on-premises intranet user wants to use a service in the AWS cloud. The service has been made highly available by two services. A high availability service will point to the active and available service by changing the IP address in the Route53 naming service.

AWS now provides the service Route 53 Resolvers for Hybrid Clouds which solves this problem.

This page explains what needs to be done to allow on-premises service consumers to get an IP address from am Route53 service. Customers who implemented Microsoft Active Directory will have to follow this blog post. The remainder of this page will deal with DNS implementations.

The problem is, that Route53 is serving only EC2 instances in a VPC. It not exposing it's service back over the VPN tunnel to the intranet.

The solution for this problem are DNS forwarders called "bind forwarders". This page explains how to create such a bind forwarder. Users will actually want to create two bind forwarders in two different availability zones to avoid a single point of failure.

The final architecture will look as follows:

Architecture for redundant bind forwarders

We will create two bind forwarders in two AWS Availabilty Zones. Corporate clients will submit their queries to the corporate DNS server. The corporate DNS server will then relay the requests to the bind forwarders. The bind forwarders will query Route53 for the final resolution.

We assume the the VPC will operate in a separate subdomain of the intranet. This will simply the routing for the corporate DNS server.

The diagram to the left shows how two application servers register or deregister themselves in Route53 win step 1 and 2. Step 3 to 5 shows how the name resolution request will be pulled from Route53.

This page describes how a bind forwarder can be configured. We assume that the we have the following network topology:

  • Intranet IP range: 10.0.0.0/8
  • Intranet domain: mycompany.corp
  • AWS VPC IP range: 10.78.0.0/16
  • AWS VPC domainname: awslab.mycompany.corp
    • IP address bind forwarder 1: 10.79.7.17
    • IP address bind forwarder 2: 10.79.8.17
    • Route53 IP address: 10.79.0.2
    • AWS regions: us-east-1

Creation of EC2 Instances

Create two EC instances. The T2 instance type is likely to be sufficient:

  • Pick the two IP addresses from above. It'll be important that the subnets are located in different Availability Zones
  • Pick Amazon Linux
  • Use Security Groups with the following inbound settings
    • ssh, tcp, port 22 open to 10.0.0.0/8
    • DNS(UDP), port 53 open to 10.0.0.0/8
    • DNS(TCP), port 53 open to 10.0.0.0/8
  • Use a certificate
  • Disk requirements minimal

Configuration of the named Service

The named service is providing the task we need.

Login a ec2-user to the instance and execute the following commands:

sudo yum install bind

Install the packages required for the named services.

Create the file /etc/named.conf. You may use the vi editor:

sudo vi /etc/named.conf

replace the content of this file with this content:

acl goodclients {
   10.0.0.0/8;
   localhost;
   localnets;
};

options {
   directory "/var/named";
   version "not currently available";
   allow-query { goodclients; };
   forwarders {10.79.0.2;};
   forward only;
};

logging{
   channel simple_log {
   file "/var/log/named/bind.log" versions 3 size 5m;
   severity warning;
   print-time yes;
   print-severity yes;
   print-category yes;
   };
   category default{
      simple_log;
   };
   category client{
      simple_log;
   };
};

zone "us-east-1.compute.internal" IN {
   type forward;
   forwarders {10.79.0.2;};
};

zone "awslab.mycompany.corp." IN {
   type forward;
   forwarders {10.79.0.2;};
};

Change all network specific values (bold) with the ones which match your configuration.

Create a log File for named Service

sudo mkdir /var/log/named
sudo touch /var/log/named/bind.log
sudo chmod ug+w bind.log

Start the Service

Execute the command:

sudo service named start

Make the service permanent and have it started after a reboot:

sudo chkconfig named on

Perform this setup for both of your EC2 instances.

 

 

Stefan Schneider Fri, 08/19/2016 - 15:09

Use the Name Services in the Intranet

Use the Name Services in the Intranet

 The final step is to make your name services in the intranet use the bind forwarders.

The intranet DNS servers will have to forward requests for the awslab subdomain to the two bind forwarders.

Individual clients may add the IP addresses of the bind forwarders to their /etc/resolv.conf file.

Stefan Schneider Fri, 08/19/2016 - 15:53