Google Compute Engine: Internal DNS server and issues with the resolving

Question

Since google Compute engine does not provides internal DNS i created 2 centos bind machines which will do the resolving for the machines on GCE and forward the resolvings over vpn to my private cloud and vice versa.

as the google cloud help docs suggests you can have this kind of scenario. and edit the resolv.conf on each instance to do the resolving.

What i did was edit the ifcg-eth0 to disable the PEERDNS and in /etc/resolv.conf i added the search domain and top 2 nameservrs my instances.

now after one instance gets rebooted..it wont start again because its searching for the metadata.google.internal domain

Jul 8 10:17:14 instance-1 google: Waiting for metadata server, attempt 412

What is the best practice in this kind of scenarios?

ty

Also i need the internal DNS for to do the poor's man round-robin failover, since GCE does not provides internal balancers.

Answer 1

OK, I just ran in to this.. but unfortunately there was no timeout after 30 minutes that got it working. Fortunatly nelasx had correctly diagnosed it, and given the fix. I'm adding this to give the steps I had to take based on his excellent question and commented answer. I've just pulled the info I had to gather together in one place, to get to a solution.

Symptoms: on startup of the google instance - getting connection refused After inspecting serial console output, will see:

Jul 8 10:17:14 instance-1 google: Waiting for metadata server, attempt 412

You could try waiting, didn't work for me, and inspection of https://github.com/GoogleCloudPlatform/compute-image-packages/blob/master/google-startup-scripts/usr/share/google/onboot

# Failed to resolve host or connect to host.  Retry indefinitely.
6|7) sleep 1.0
log "Waiting for metadata server, attempt ${count}"

Led me to believe that will not work.

So, the solution was to fiddle with the disk, to add in nelasx's solution: "edit ifcfg-eth and change PEERDNS=no edit /etc/resolv.conf and put on top your nameservers + search domain edit /etc/hosts and add: 169.254.169.254 metadata.google.internal"

To do this,

1. Best to create a snapshot backup before you start in case it goes awry

2. Uncheck "Delete boot disk when instance is deleted" for your instance

3. Delete the instance

4. Create a micro instance

5. Mount the disk

   sudo ls -l /dev/disk/by-id/* # this will give you the name of the instances

   sudo mkdir /mnt/new

   sudo mount /dev/disk/by-id/scsi-0Google_PersistentDisk_instance-1-part1 /mnt/new

where instance-1 will be changed as per your setup

6. Go in an edit as per nelasx's solution - idiot trap I fell for - use a relative path - don't just sudo vi /etc/hosts use /mnt/new/etc/hosts - that cost me 15 more minutes as I had to go through the: got depressed, scratched head, kicked myself cycle.

7. Delete the debug instance, ensuring your attached disk delete option is unchecked

8. Create a new instance matching your original with the edited disk as your boot disk and fire it up.

Answer 2

As mentioned at https://cloud.google.com/compute/docs/networking:

Each instance's metadata server acts as a DNS server. It stores the DNS entries for all network IP addresses in the local network and calls Google's public DNS server for entries outside the network. You cannot configure this DNS server, but you can set up your own DNS server if you like and configure your instances to use that server instead by editing the /etc/resolv.conf file.

So you should be able to just use 169.254.169.254 for your DNS server. If you need to define external DNS entries, you might like Cloud DNS. If you set up a domain with Cloud DNS, or any other DNS provider, the 169.254.169.254 resolver should find it.

If you need something more complex, such as customer internal DNS names, then your own BIND server might be the best solution. Just make sure that metadata.google.internal. resolves to 169.254.169.254.