# Learning CoreDNS reading notes

# Chapter 1: Introduction

CoreDNS is a forked of Caddy, written in Go. It uses `Corefile` for simple configuration. It uses a system of plugins for features.

## Limitations
CoreDNS doesn't support recursion. It relies on other `forwarders` (DNS servers)


# Chapter 2: A DNS Refresher

## What is the DNS?
A naming system that maps names to other data, such as IP addresses, mail routing... It's one of the largest distributed databases in the world.

Works in client-server model (resolvers and name servers).

## Domain Names and the Namespaces

## Domains, Delegation and Zones
A zone is a domain minus the subdomains that have been delegated elsewhere.

## DNS Servers and Authority
DNS servers's responsibilities: answering queries and querying other DNS servers.

DNS servers load zone data from `zone data files` (`master files`). `primary DNS server` of a zone: a DNS server that loads information about a zone from a zone data file.

`zone transfer`: DNS servers (secondary ) load zone data from other DNS servers (master DNS server). The copy of zone data file can be copied to disk to load after secondary servers restart.


Both primary and secondary are `authoritative` for the zone. They can answer any query for a domain name in the zone definitely.

## Resolvers
Clients of the DNS. They queries DNS servers, receive answers within TTL (retransfer or change DNS servers). They can also do caching.

## Resolution and Recursion
DNS servers need a `hint` to direct them where to start. The information of root is provided by the `root hints` (either compiled into DNS server or contained in a file). The hints are NS records give the domain names of the DNS servers authoritative for the root zone.


Root servers will return the list of DNS servers authoritative for the top-level zone: `referral`. The DNS server continues following referrals until it reaches the DNS servers authoritative for the domain name in the query.


Only the first DNS server performs recursion. It's because the resolvers send the recursive requests to it, and it only sends non-recursive requests to the referral DNS servers.

If the first DNS server is configured to use the second as `forwarder`, it will send recursive requests.

Forwarders are used to resolve domain names in the internet's namespaces. The internal DNS server are configured to use a DNS server with internet connection as a forwarder. The internal DNS server only checks its authoritative zone data and cache before consulting a forwarder.

## Resource Records
Records in master file have the following format

[NAME] [TTL] [CLASS] TYPE RDATA

### NAME
This can be a FQDN, ending in a dot, or a relative domain name.


### TTL
How long the DNS server caches the record. 32-bit number of seconds.


### CLASS
Almost always IN, for internet. CH (ChaosNet) and HS (Hesiod) but would rarely be used.

## Resource Record Types

### A
Domain name -> IPv4. RDATA is a single IPv4 address


### AAAA
Domain name -> IPv4. RDATA is single IPv6 address


### CNAME
Alias from one domain name to another. RDATA is the canonical name


Some rules:
- The alias domain can't have other record types.
- Domain name of the zone can't own a CNAME, by definition it must own a start of authority record (SOA)
- There are limit of recursive CNAME, don't create too much loop.

### MX
MTA will look for MX records first before going to A or AAAA records when sending emails. RDATA is preference (0-65535) + the mail exchanger of the domain.


### NS
The name server for a given zone. RDATA is the domain name of a DNS server authoritative for the zone


The NS records or a domain name usually appear in 2 zones (itself and its parent zone)

### SRV
SRV record provies a layer of abstraction between domain names and the servers for any kind of services (similar to MX)

_service._protocol.domainname

Clients interested in a particular service running over a particular protocol at a certain domain name would concat the service, proto, domain name to form a new domain name and lookup the SRV records for that domain name. The `_` to avoid duplicating with real domain names.

RDATA has 4 fields:
- Priority
- Weight: if same priority, weight is evaluated.
- Port
- Target: domain name of the server


### PTR
IP -> domain names

DNS requires two special namespaces. One is `in-addr.arpa`, used to reverse-map IPv4 addreses to domain names. The other is `ip6.arpa` used for IPv6.

RDATA is just a single domain name.


### SOA
Provides summary information about a zone. Only 1 SOA per zone. RDATA has 7 field:
- MNAME: domain name of the primary DNS server for the zone
- RNAME: email address of the person in charge. @ -> .
- Zone's serial number, 32 bit unsigned
- Zone's refresh interval
- Zone's retry interval
- Zone's expiration interval
- Zone's negative-caching TTL


The serial number, the intervals are related to zone transfer


NOTIFY messages: sent from master DNS servers to secondaries to inform zone's data changed.

Negative-caching TTL: how long other DNS servers can cache negative responses from this zone's authoritative DNS servers. Negative responses:
- No such domain name
- No such data

Negative caching is helpful for preventing DNS servers overloaded but the TTL should not be too long.

# Chapter 3: Configuring CoreDNS
Using `Corefile` (similar to `Caddyfile`)

## Corefile syntax



# root block
. {
  directives...
}

# custom port for server
.:1053 {
  directives...
}

# DNS over TLS (DoT)
tls://foo.example {
  directives...
}

# DNS over gRPC
grpc://bar.example {
  directives...
}

When CoreDNS receives a query, it examines the `Corefile` to find an applicable server block. The longest match will get the config.

## Plug-ins
The order of plugin processing is fixed, not related to the order of plugin in Corefile.

### Root
Specifies the current working directory

. {
    root /etc/coredns/zones
}

### File
The File plugin configures the server as the primary DNS server for the zones.


If ZONES is omitted, the file will be read as the zone data file for the zone in the server block. The ZONES must fall within the zone in server block.

`transfer`: outbound transfers of the zone (to secondary DNS server)

`reload`: period of rechecking the zone data file, check for SOA serial number


### Secondary

CoreDNS servers do not store data for secondary zone in a backup file. It needs to retransfer zone data from it's master DNS servers each restart. And it doesn't support (IXFR), incremental zone transfer -> full transfer.


### Forward

Queries matching FROM are forwarded to TO.

CoreDNS uses an internal DNS healthcheck system to determine the health of forwarders, if all fwders unhealthy -> CoreDNS thinks it's healthcheck system failed and pick a random fwder.

### Cache

### Errors

Intructs CoreDNS to log errors. They are sent to standard output.



### Log
Intructs CoreDNS how to log. Default to stdout.



## Common Configuration Options
A few common configuration in other plugins.

### fallthrough
By default: CoreDNS provides response for queries for a zone that a plugin has authority. If the domain does not exist, it returns the DNS response code NXDOMAIN. If domain exists but no data, returns empty answer (NODATA).

We might want to use another plugin to answer the query, use `fallthrough`.

### tls
This option enables client-side TSL certificates. It's not the `tls` plug-in, which is used to configure CoreDNS server's TLS certifcates.

Without parameters, the TLS client should verify server's client cert using standard certificate authorities.

`tls CERT_FILE KEY_FILE`: server cert will be verified using standard cert authorities on the system.

`tls CERT_FILE KEY_FILE CA_FILE`: server cert will be verified using CA file.

### transfer to
Enables outbound transfers of the zone

## Sample DNS servers configuration
### Caching only server

. {
  forward . 8.8.8.8 8.8.4.4
  cache
  errors
  log
}

### Primary DNS server

foo.example {
  root /etc/coredns/zones
  file db.foo.example
  errors
  log
}

# If you want your DNS server to handle recursive queries, too,
# you'll need an entry like the following.  If it's authoritative-
# only, omit it

. {
  forward . 8.8.8.8 8.8.4.4
  cache
  errors
  log
}

### Secondary DNS server

(logerrors) {
  errors
  log
}

bar.example {
  transfer from 10.0.0.1
  import logerrors
}

# It can be used as primary server and cache server

# Chapter 4: Managing Zone Data
Many different options to store zone data.

## The file Plugin
Syntax:

file DBFILE [ZONES...] {
  transfer to ADDRESS...
  reload DURATION
  upstream [ADDRESS...]
}

Must specify the `transfer` to allow zone transfers.


## The auto Plugin
A clever way to load a large number of zones from multiple zone data files at once.

Syntax:

auto [ZONES...] {
  directory DIR [REGEXP ORIGIN_TEMPLATE]
  transfer to ADDRESS...
  reload DURATION
}

This plugin tells CoreDNS to scan the directory for files matching pattern `db.*` (default). Each file is a zone data file of the origin in `db.`

### Using the plugin with Git
Using git-sync to reload the zone directory


## The hosts Plugin
Used to configure CoreDNS to generate zone data from a host table (e.g `/etc/hosts`). Must be in this format

<IP> <canonical name> [aliases...]

The plugin will create A, AAAA and PTR records for each entry in the host table.

Syntax

hosts [FILE [ZONES]] {
  [INLINE]
  ttl SECONDS
  no_reverse
  reload DURATION
  fallthrough [ZONES...]
}

FILE specifies the host file, by default `/etc/hosts`.

Zones in this plugin are not complete zones, they don't have SOA records, so can't be transferred.

`[INLINE]` for inline host entries

`no_reverse` inhibits the creation of PTR records from host table entries.

## The route53 Plugin
CoreDNS acts as a secondary DNS server, with the master DNS server is Route53.

Syntax


# Chapter 5: Service Discovery
## Solving the Service Discovery Problem
`Service registration and discovery`: the controller or orchestrator launching the process can make API call to register it.

Because the data would change rapidly, to avoid the harm of TTL, service discovery solutions provide a method to push data to clients. DNS does not support push based.

CoreDNS relies on gRPC to push data to client (experiment).

## Service Discovery with CoreDNS and etcd
`etcd`: developed by CoreOS, using `quorum` for consistency.

### The etcd Plugin
This plugin reads data from `etcd` using `etcdv3 API`

Data is stored in `etcd`, read on demand and cached from CoreDNS, which makes it satateless.

**SkyDNS Message format**
services are stored in etcd using the SkyDNS message format

The keys are service names

{
  "host": "192.0.2.10",
  "port": 20020,
  "priority": 10,
  "weight": 20
}

Except host field are string, other are integer for format parsing.

The message need to be stored in `etcd` under a specific key. The format of the key:

`users.services.example.com` -> `/skydns/com/example/servcies/users` (`skydns` is configurable)


Can also use query to SRV record.

Complete syntax of `etcd` plugin

etcd [ZONES...] {
  stubzones
  fallthrough [ZONES...]
  path PATH
  endpoint ENDPOINT...
  credentials USERNAME PASSWORD
  upstream [ADDRESS...]
  tls CERT KEY CACERT
}

`stubzones`: deprecated, replaced by `forward`

`fallthrough`: pass queries to plugins

`path`: override default `/skydns`

`endpoint`: list of etcd API endpoints, e.g: `https://1.2.3.4:2379`

`credentials`: etcd username/password

`upstream`: deprecated

`tls`: TLS client for connecting to `etcd`

### Other Service Discovery Options
We can use a relational database instead of `etcd`. For example `pdsql`

Using host table file is also an possible option.

## Service Discovery and Container Orchestration
The registration is handled by container orchestration such as Kubernetes.

# Chapter 6: Kubernetes

## Kubernetes DNS Specification
DNS is an `add-on` in K8s but most distribution provide it.

All records fall under a single domain, the `cluster domain`, often is `cluster.local`. Vanilla K8s cluster allows to change the domain, cloud ones don't.

The specification requires for each cluster IP, there is an A record with name derived from the service name and namespace: `service.namespace.svc.cluster-domain`. There will also be a corresponding PTR record for the cluster IP.

For headless service, there are A records for every endpoint IP for the service. This allows to use client-side load balancing.


There are SRV records for each endpoint address and named-port combination

dig -t srv _http._tcp.headless.default.svc.cluster.local.


The `ADDITIONAL SECTION` contains A records reffered to by the SRV record's targets. PodSpec can use `hostname` and `subdomain` to overwrite the default value.

hostname: myhost
subdomain: headless


In Daemonset, `hostname` is created based on ordinal number (0,1,2,...)


## CoreDNS Integration
CoreDNS uses the `kubernetes` plugin. It creates a watch on the Services and Endpoints resources, caches that data (in-memory). Fast because it doesn't need to request the API server for each query.


This cache behavior is for K8s resources caching, not the `cache` plugin which is DNS cache.

`kubernetes` plugins watches `Services` and `Endpoints` resource. Endpoints is troublesome because it contains both ready and not ready backends. And the more backend an Endpoint has, the higher chance that it will change, which makes the watch API to reload the whole Endpoints set -> overload to CoreDNS instance.

=> Consider turn off Endpoints watch with `noendpoints` option if do not need to support headless services.

## Default Configuration
Most K8s deployment cover recommendations of the CoreDNS team for `Corefile`

.:53 {
  errors
  health
  kubernetes CLUSTER_DOMAIN REVERSE_CIDRS {
    pods insecure
    upstream
    fallthrough in-addr.arpa ip6.arpa
  }
  prometheus :9153
  forward . UPSTREAMNAMESERVER
  cache 30
  loop
  reload
  loadbalance
}

`health`: health-check endpoint for kubelet, HTTP 8080, `/health`

`kubernetes`:
- specifies the cluster domain, `REVERSE_CIDRS` for K8s Service CIDR. If want to resolve PTR queries for service endpoints, also need pod CIDRs in this list. It's because service and pod CIDRs can't be discovered through API.
- `pods insecure`: fallback support for a deprecated of `kube-dns`
- `upstream`: allows resolve CNAME by sending the queries back to CoreDNS itself, it's default in Kubernetes later than 1.4
- `fallthrough in-addr.arpa ip6.arpa`: fallthrough for PTR queries, pass to the next plugin which is `forward`

`forward . UPSTREAMNAMESERVER`: forward for root zone

`cache 30`: not really necessary as explaining before, `kubernetes` plugin already handled the caching part for resources

`loop`: detect query loop and exit

`reload`: hot reload Corefile

`loadbalance`: randomly shuffles A/AAAa records in the response.

## Stub Domains and Federations


`STUBDOMAINS`: populate additional server blocks to resolve specific domains instead of forwarding to the UPSTREAMNAMESERVER

`FEDERATIONS`: supports K8s federation v1 names (deprecated)

## Cluster DNS Deployment Resources
Other K8s resources that needed for CoreDNS to be up and running: container image, replicas, deployment details...

### RBAC


`pods` and `nodes` privilege may not be needed if using the default Corefile configuration, but may be needed in future usage, though, can be safely removed

`namespaces`: whether returns NXDOMAIN or SUCCESS with no data when query domain in a ns. Also necessary when querying wildcard queries.

### Service
CoreDNS uses a standard ClusterIP service


The ClusterIP is fixed at bootstrap and passed into kubelet to create the `resolv.conf` file for the pods.

Note: Prometheus uses the metrics port to determine the port to scrape from pods, not through service.

### Deployment
`Default` dnsPolicy (used in CoreDNS): is not the default dnsPolicy, it means to use the host node's DNS configuration.

The default dnsPolicy for pods is `ClusterFirst` -> use the CoreDNS for name server.

Corefile is mounted as a configmap.

Scale testing shows CoreDNS needs < 170Mi memory for handling a 5000 node cluster.


## Autoscaling
Default is 2 replicas. To handle autoscale, many distributions deploy a dns-autoscaler pod, cluster-proportional autoscaler. For example 1 pod per 256 cores or 1 pod per 16 nodes.

We can also use HPA to scale CoreDNS.

CoreDNS also publishes a metric name `coredns_health_request_duration_seconds` = how long CoreDNS responds to health probe in `health` plugin.

## A Better Configuration

CLUSTER_DOMAIN REVERSE_CIDRS {
  errors
  health
  kubernetes
  ready
  prometheus :9153
  loop
  reload
  loadbalance
}

. {
  errors
  forward . UPSTREAMNAMESERVER
  prometheus :9153
  cache
  loop
}

Queries for the K8s cluster will not be cached (as it's useless).

A drawback: kubernetes plugin is loaded many time for each zone in server block.

Some plugin in kubernetes plugin are dropped because deprecated or not needed, including the `forward`.

## The kubernetes Plugin
Some extra options allow CoreDNS to run externally to the cluster.

Full syntax


`ignore empty_service`: return NXDOMAIN instead of no data for service with no backend.

`endpoint_pod_names`: changes how endpoint names are determined

`noendpoints`: disables endpoint records

`pods`: pod options

## The CoreDNS Extensions
Should be careful to use since they can affect workload portability

### Pod Options
In K8s DNS specification, pod records are deprecated. Pod options offer 3 ways to handle pod name requests

`pods disabled`: default of CoreDNS, return NXDOMAIN

`pods insecure`: default of K8s configuration, maintains backward compatability

`pods verified`: CoreDNS watches on pod resources, return IP for `a-b-c-d.namespace.pod.cluster.local`. It could introduces performance and scaling issues.

### Wildcard Queries
It's **distinct** from the concept of wildcard records in DNS. It's a special feature for service discovery

Example, but not really useful this case


Another example, get all endpoints of a service


### Autopath and the Dreaded ndots:5
To allow using short names `service` instead of `service.namespace.svc.cluster.local`, kubelet sets search path in a pod's resolv.conf to the following domains, in order:

1. `<namespace>.svc.cluster.local`: allows name like `<service>`
2. `svc.cluster.local`: allows name like `<service>.<namespace>`
3. `cluster.local`: ...
4. The host search path, usually a couple of domains

Works well when resolving in-cluster names, however, trouble in resolving external names. For the search path to work, kubelet also sets the `ndots` option to five: any domain name fewer than five dots is considered to be a possible relative domain.


For a single query, 5 `failed queries` was made, could be increase the load on CoreDNS.

Some solutions: use FQDN for external names, manually config `dnsConfig` on pod spec. Another solution is to use node local cache, a small caching-only build of CoreDNS on each node, this could increase resource usage on node and not HA compatible.

CoreDNS has another solution: `autopath`. CoreDNS figure out the search path on the server side, it will iterate through the queries internally, no network back forth involved. The biggest downside is the scalability, to figure out search path, CoreDNS needs to know the pod's NS, which requires turning `pods verified`

### Zone Transfer Support
A good debug tool because we can see all k8s records in a single request.

Enable it: `transfer to *` in `kubernetes` plugin.


### Exposing Services Externally
Use externalDNS, or simpler plugin `k8s_external`

Configuring an external zone:

k8s_external services.example.com {
  apex services
}

For external usage, we should use another instance of CoreDNS to avoid harmful load that makes the primary CoreDNS instances down.

### Modifying the Available Records
Tweak the configuration of k8s plugin to change the way records are presented, for example, in multitenant use case, a separate CoreDNS handles records for certain ns.

We can use `namespaces` and `labels` options for this use case, `namespaces` accepts a list of ns, then modify kubelet to set the NS to the separated CoreDNS server. `labels`: label selector for a list of ns.

# Chapter 7: Manipulating Queries and Responses

## The template Plugin

Creation of answers to PTR queries, without having to actually write them all out in a zone file.


We can also use `template` to block names that we do not want queried


## The rewrite Plugin
Based on the same name plugin of Caddy. It can modify requests (record type, name, class, EDNS0 options) or responses (TTL, domain names,...)

Syntax


`continue` and `stop`: how multiple rules are handled

`FIELD`: what component of the request you want to modify, valid values: `type`, `class`, `name`, `dns0` or `tll`

Use case: use the same TLS cert for different names to access a site, similar to CNAME but transparent to clients.

Example: internal cluster request to `api.example.com` must go out and in to get the cert.

Solution:



The `rewrite` plugin also works with regex to allow multiple names in a domain.

The answer will contain the rewritten domain, which could cause security block, we can use `answer name` to rewrite it to the original domain name.


### Using the rewrite Plugin for ENDS0 Options
EDNS0 (Extension Mechanisms for DNS) allows additional options to be included in DNS req/res

Syntax


Can be used for specific use case for `edge` CoreDNS, deployments of CoreDNS in home or branch office, provides CoreDNS to local devices.

### Multiple rewrite Rules


Rules are processed in order.

## The metadata Plugin
Allows plugins to publish and consume metadata from other plugins.


This config reads the EDNS0 local option 0xffed and publish it under metadata name `{/metadata_edns0/client_id}`.

## Signing Responses with DNS Security Extensions
DNSSEC allow administrators of DNS zones to sign records and admins of recursive DNS servers validate signed records -> prevent DNS cache poisoning.

### Managing a DNSSEC-Signed Primary Zone
If we use CoreDNS as primary DNS server (with `auto` or `file` plugins), use tools like `dnssec-keygen` and `dnssec-signzone` to generate keys, sign zones and manage the signed zones.


The command generated a Zone-Signing Key (ZSK) pair. Private key is used to sign records, public key is used to validate records.


The second command generated a Key-Signing Key (KSK) pair. Private key is used only to sign keys (ZSK) in zone, main reason is for key rotation of ZSK

Then we can put the key files into zone database file for configurating the zone.


This command will create a new signed zone database file and can be loaded into CoreDNS

### On-the-Fly DNSSEC Signing with the dnssec Plugin
Using `dnssec` plugin to sign records on the fly.


`cache_capacity`: size of cache for storing generated signatures, default is 10000 signatures.


# Chapter 8: Monitoring and Troubleshooting

## The prometheus Plugin
Syntax

prometheus [ ADDRESS ]

HTTP server with a handler for `/metrics`

## The log Plugin
Example log message

## The dnstap Plugin
`dnstap` is a flexible, structured, binary log format for DNS, using minimal resources.

Syntax

dnstap SOCKET [full]

Example

dnstap tcp://127.0.0.1/8053
dnstap unix:///tmp/dnstap.sock
dnstap /tmp/dnstap.sock

`SOCKET` is a TCP or Unix domain socket that CoreDNS will write dnstap information

`full`: send the contents of the query

The dnstap information can be read and parsed by a dnstap program like `golang-dnstap`.


## The errors Plugin
Already described in previous part


## The trace Plugin
Allow to integrate CoreDNS with distributed tracing tools: `Zipkin`, `Datadog`,...


By default, it will use Zipkin and send traces to `localhost:9411`

Reduce tracing for performance, one of every N requests


Syntax


`client_server`: if specified, client and server share the same span

## The debug Plugin
By default, if CoreDNS crashes, it will restart on its own.

`debug` instructs CoreDNS not to recover from a crash and send `log.Debug`/`log.Debugf` to standard output, which help to diagnose the issue.


DON'T USE IT IN PRODUCTION