Thesis: Monitoring DNS registry mutations with an Ensemble Anomaly Detection Framework

This challenge becomes particularly interesting as operational environments continue to evolve. Across many sectors, organisations are increasingly adopting distributed, cloud-integrated and externally managed services to improve scalability, flexibility and operational efficiency. Similar modernisation efforts can also be observed within the DNS industry. For example, SIDN has discussed the ongoing modernisation of the .nl registration system and the use of hybrid cloud architectures as part of its long-term infrastructure strategy. Similar developments can be observed elsewhere. DNS Belgium migrated its domain registration platform to a hyperscaler in 2017 and has subsequently announced plans to migrate critical registry services away to a European cloud provider, illustrating the evolving role of cloud infrastructure in modern infrastructure deployments.

While such developments provide important operational benefits, they also increase the complexity of operational workflows. Modern registry processes increasingly involve APIs, automation platforms and interactions between multiple systems. As a result, understanding whether a particular registry mutation is expected or unusual becomes an increasingly relevant operational challenge.

That observation formed the starting point of my master's thesis. Rather than focusing on how attacks enter a system, the research investigates whether machine learning can help identify suspicious DNS registry mutations before they become part of the authoritative DNS state.

From prevention to detection

DNS registries already employ extensive security and operational controls designed to protect the integrity of registry data and ensure the correct functioning of registration workflows.

However, those mechanisms are designed primarily to verify that changes are valid, authorised and consistent with operational procedures. They are generally not designed to determine whether a change appears unusual when viewed in the context of historical behaviour, infrastructure relationships and mutation patterns across the registry ecosystem.

Consider a malicious mutation update that is technically valid, submitted through a legitimate account and fully compliant with operational procedures. From an operational perspective, there may be no immediate indication that anything is wrong. Yet the change could cause unintended effects that potentially lead to downtime or enabling an attacker to intercept traffic for that domain.

The research described in this blog explores whether machine learning can provide an additional layer of visibility by learning what normal registry behaviour looks like and highlighting mutations that deviate from those patterns before they become part of the authoritative DNS state. Rather than replacing existing controls, the goal is to complement them with behavioural monitoring that can help prioritise potentially suspicious changes for further investigation.

Understanding the threat landscape

An important challenge at the start of the research was determining what suspicious DNS registry behaviour actually looks like, as there are virtually no publicly available datasets containing examples of suspicious or malicious registry mutations.

To address that challenge, I developed a set of threat scenarios based on academic literature, internal threat analyses, and expert interviews. The resulting scenarios covered a range of registry-related abuse patterns, including unauthorised delegation changes, glue record manipulation, DNSSEC misuse, registrar compromise and coordinated attacks involving multiple domains.

Although those scenarios differ technically, they share a common characteristic: they ultimately manifest themselves as unusual changes to registry data. That observation influenced the overall design of the framework, which focuses on identifying suspicious state transitions and behavioural patterns rather than specific attack techniques.

Figure 1 provides an overview of the proposed anomaly detection framework. Starting from registry mutation data, the pipeline combines feature engineering, graph construction, anomaly detection and explainability components to identify mutations that warrant further investigation.

Figure 1: Overview of the proposed anomaly detection framework. The process begins with registry mutation sampling and state reconstruction, followed by feature extraction and DNS graph construction. Isolation Forest and GraphSAGE then generate complementary anomaly scores, which are combined through an ensemble decision layer and thresholded to identify anomalous events. The final phase generates explainable alerts to support analyst investigation of suspicious DNS registry mutations.

Combining 2 perspectives

An important observation during the research was that DNS registry data is inherently relational. Registry mutations do not occur in isolation; instead, they involve interconnected entities.

Viewed individually, many of the entities in question appear entirely normal. However, the relationships between them can reveal patterns that are difficult to identify using traditional feature-based approaches alone.

That observation motivated the use of graph-based representation learning. The registry ecosystem was modelled as a heterogeneous graph containing domains, name servers, registrars, IP subnets and DNSSEC-related entities connected through their operational relationships.

At the same time, the research demonstrated that some anomalies are more visible through behavioural and statistical signals than through infrastructure relationships alone. That ultimately motivated the use of an ensemble approach combining a graph neural network with an isolation forest. Together, those complementary perspectives allow the framework to identify both relational and behavioural anomalies that may warrant further investigation. Figure 2 illustrates how the graph neural network and isolation forest contributed differently across the evaluated attack scenarios.

Figure 2: Relationship between GNN and isolation forest anomaly scores. The distribution of points shows that high anomaly scores produced by one model do not always correspond to high scores produced by the other, indicating that the two models capture different aspects of anomalous behaviour. This complementary behaviour motivates their combination within the ensemble framework.

As Figure 2 demonstrates, the 2 models do not capture exactly the same anomaly patterns, providing a strong motivation for combining both approaches within a single ensemble framework.

The outputs of both models are combined into a single anomaly score, as can be seen in Figure 1, Phase 4 (Decision Layer). This ensemble approach allows the framework to benefit from both relational and feature-level perspectives, providing broader anomaly coverage than either model alone.

When suspicious activity looks normal

Figure 3: Distribution of ensemble anomaly scores with synthetic attack markers. The black line denotes the anomaly threshold. The red marker corresponds to the glue manipulation scenario, whose score lies close to the decision boundary and highlights the difficulty of detecting subtle infrastructure-level attacks. Green markers correspond to attack scenarios that are consistently assigned high anomaly scores and are clearly distinguishable from normal registry behaviour.

Figure 3 shows the anomaly score distribution produced by the framework, together with the injected attack scenarios. Most attacks are clearly distinct from normal registry mutations and appear among the highest-scoring observations.

One result, however, stood out. Glue record manipulation proved considerably more challenging to detect than any other implemented attack scenario. As visible in Figure 3, most attack scenarios cluster near the highest anomaly scores (green vertical lines), whereas the glue manipulation scenario remains much closer to normal operational behaviour (red vertical line).

Unlike many attacks that significantly alter relationships between domains, name servers and infrastructure, glue manipulation can preserve much of the surrounding DNS structure. Domains continue pointing to the same name servers, registrars remain unchanged and DNSSEC relationships may remain unaffected. From the perspective of the graph, relatively little appears to change.

At the same time, changes to name server infrastructure are not uncommon in legitimate registry operations. As a result, malicious glue modifications can closely resemble normal administrative activity, making them difficult to distinguish from benign changes using behavioural and structural signals alone.

That finding highlights an important lesson: the most difficult anomalies to detect are not always the most disruptive or complex. In practice, attacks that closely mimic legitimate operational behaviour may present one of the greatest challenges for anomaly detection systems. Although that finding highlights the limitations of anomaly detection, additional (existing and deployed) security mechanisms such as DNSSEC provide complementary layers of protection that help reduce the likelihood and impact of a successful attack.

A new perspective on registry monitoring

The results demonstrate that suspicious DNS registry mutations can be detected through behavioural and relational analysis, even in the absence of labelled attack data. By combining threat-model-driven evaluation with unsupervised machine learning, the framework was able to consistently identify and prioritise the majority of implemented attack scenarios.

An important observation was the complementary nature of the ensemble architecture. While the graph neural network captured relationships between domains, name servers and infrastructure, the isolation forest contributed sensitivity to behavioural and statistical anomalies. Together, the 2 perspectives provided broader anomaly coverage than either approach individually.

The research also emphasised the importance of explainability within critical infrastructure environments. Identifying a suspicious mutation is only part of the challenge; operators must also understand why it was flagged in order to support investigation and operational decision-making.

Perhaps the most important outcome of the research is the concept of a pre-zone behavioural risk filter. Rather than replacing existing security controls, the framework introduces an additional layer of visibility that evaluates mutation behaviour before changes are incorporated into authoritative DNS zones.

Looking ahead, future work could explore dynamic graph learning techniques, integration of additional operational context such as passive DNS and BGP data, and complementary DNSSEC validation mechanisms. More broadly, the research illustrates how graph-based machine learning and behavioural analysis can complement existing registry security practices and improve visibility into complex DNS infrastructure relationships.

Overall, the research demonstrates that behavioural anomaly detection is a viable approach for monitoring DNS registry mutations. The proposed framework successfully detected 6 of the 7 implemented attack scenarios derived from the threat model, showing that machine learning can provide meaningful additional visibility into suspicious registry activity. Even though this research was conducted based on synthetic data, we believe the insights following from this research are valuable for operational systems. Rather than replacing established security measures, the framework is intended to operate as an additional defence-in-depth layer that helps identify potentially risky mutations before they become incorporated into authoritative DNS zones. The experimental evaluation was conducted on approximately 24,000 registry mutations using a single GPU, suggesting that the computational requirements are manageable and that near real-time deployment may be feasible within existing registry publication workflows. However, further evaluation on complete registry datasets is needed to assess scalability and operational performance at full registry scale. Future work should also investigate mechanisms for adapting to evolving registry behaviour through incremental learning, periodic model updates, or dynamic graph learning approaches, reducing the need for frequent retraining while maintaining detection effectiveness in continuously changing DNS environments.