The following illustration shows how a service would look-up another service by its service name it has been assigned in your Nomad Job. The service name look-up request is made to the DNSMASQ service running on the Linux host the Nomad Job is running on, which is then forwarded to Consul Servers(Service Discovery and DNS). Take note that the diagram does not fully illustrate a complete Nomad Cluster, it is a high-level logical depiction on how service to service name resolution works between Nomad service jobs.
The Consul client(running along side Nomad client on the same host) will help register new Nomad Job services that are deployed to the Consul Cluster. So anytime a service(app code, api or nomad job file spec) needs to communicate with another service, the service name lookup is performed by DNSMASQ which is then forwarded to Consul.
DNSMASQ
DNSmasq is fundamentally a lightweight DNS Resolver/Forwarder which can forward service or host name lookups to upstream DNS Servers and Service Mesh platforms(like Consul!), which can communicate and listen on custom ports. In this case DNSMASQ will forward all *.consul DNS lookups to the Consul DNS Servers(which is the Consul Server Cluster) on port 8600. All other name resolution requests will be forwarded to upstream DNS Servers.
DNSMASQ Linux Install and Configuration
DNSMASQ is installed on the Hashicorp NOMAD client/worker nodes only. You can download the dnsmasq source from here to compile for most linux distros. The following install steps are for Rocky 9/CentOS/Redhat Linux distributions;
- Install DNSMASQ
sudo dnf install dnsmasq dnsmasq-utils
sudo systemctl enable dnsmasq
sudo systemctl start dnsmasq- Make a backup of the original default dnsmasq config
sudo cp /etc/dnsmasq.conf /etc/dnsmasq.conf-ORIG- Edit the /etc/dnsmasq.conf file with the following base configuration
# Configuration file for dnsmasq.
#
# Listen on this specific port for DNS.
# Setting this to zero completely disables DNS function,
# leaving only DHCP and/or TFTP.
port=53
# IP Address of the upstream DNS Server to forward on all non .consul queries
# Enter list of IP Address's of Global DNS resolution.
server=XXX.XXX.XXX.XXX
# Add other name servers here, with domain specs if they are for
# non-public domains.
# Enter in the IP Address of the Consul Servers, 2 or 3, along with Consul DNS port 8600
# All *.consul service/hostname lookups will be forwarded to the the Consul Servers to deal with.
# CONSUL DNS
server=/consul/XXX.XXX.XXX.XXX#8600
server=/consul/XXX.XXX.XXX.XXX#8600
# If you want dnsmasq to change uid and gid to something other
# than the default, edit the following lines.
user=dnsmasq
group=dnsmasq
# If you want dnsmasq to listen for DHCP and DNS requests only on
# specified interfaces (and the loopback) give the name of the
# Repeat the line for more than one interface.
#interface=
# Listen only on localhost by default
interface=lo
# The second address after 127.0.0.1 is the interface IP address of the Nomad client host
listen-address=127.0.0.1,XXX.XXX.XXX.XXX
# If you want dnsmasq to provide only DNS service on an interface,
# configure it as shown above, and then use the following line to
# disable DHCP and TFTP on it.
# Check interface name of the client host
no-dhcp-interface=eth0
# To listen only on localhost and do not receive packets on other
# interfaces, bind only to lo device. Comment out to bind on single
# wildcard socket.
bind-interfaces- Update Nomad Client Linux Hosts /etc/resolv.conf
- Remove all nameservers from /etc/resolv.conf and Add IP address of client agent/worker to /etc/resolv.conf as a nameserver, effectively pointing back to itself, because DNSMASQ service is running locally. dnsmasq service will forward all non .consul lookups to external/upstream DNS gateways configured in the dnsmasq.conf file.
# /etc/resolv.conf
nameserver 123.123.123.123Service to Service Communications on Nomad Cluster Configured with Consul
The above simple base configuration should get you started with cluster service to service name discovery/lookup without having to figure out IP Addresses or hostnames of where the service is running on. All Nomad jobs submitted to the Hashicorp Nomad/Consul cluster will now use the local client host dnsmasq service to resolve service names for all jobs internally within the cluster. Consul will load-balance the service discovery with ACL/s and “service intentions(traffic control)” with mTLS for encryption.
For further information on Hashicorp’s Consul Service Mesh technology, you can read more HERE