DSI & Panopticon

The CoCCA Domain Security Initiative (DSI) is a public benefit project to help ccTLD managers understand and mitigate security issues involving domains in the TLDs they manage. The project does not analyse data on very large ccTLDs or generic / new ICANN TLDs.

Panopticon and Arke are the names of the two tools developed for the CoCCA DSI, the tools are GDPR compliant.

Panopticon and Arke are template driven - sources, indicators, proxy locations are all configurable per TLD (or group of TLDs) as requested by the ccTLD manager.

Panopticon reports are reproducible and include all the raw data.

For information email: dsi@coccaregistry.org


DSI & DAAR

The ICANN Domain Abuse Activity Reporting (DAAR) project is a system to study, compare and report on security threats in TLDs. DAAR and the DSI are similar in some respects, but they have different objectives and differ in several substantive ways:

  • The DSI validates each URI four times a day using DNS lookups and http/https tests via two different geo located proxy networks. DAAR was designed for both ICANN TLDs and ccTLDs, for reasons related to testing at scale, DAAR makes no attempt to validate data provided by third parties.
  • The DSI tools focus on notification regarding active, verified threats. DSI Reports differentiate between delegated, undelegated, and non-existent domains. DAAR reports count un-delegated and non-existant domains that appear in blocklists as "abuse domains".
  • The DSI tracks, validates, and reports on domains in the registry, subordinate hosts created externally, and unique URLs. DARR does not.
  • The DSI validates phishing URI four times daily. Threats are only excluded from statistics when they cease to be reachable. DAAR automatically closes phishing cases the day they are reported.

The CoCCA DSI uses publicly available data, it does not require zones, or a data sharing agreement. The DSI does not attempt to rank or compare TLDs.

ICANN DAAR Methodology Paper (pdf).

Process

  • Automate the daily collection of malicious Uniform Resource Identifiers (URIs) for small and medium sized ccTLDs from 30 threat Intelligence feeds.
  • Analyse URI host names to distinguish between domains registered in the ccTLD registry and subordinate domains created outside to the registry.
  • Query the applicable DNS servers to see if the domain is delegated and active (nxdomain, NS, A/AAAA, MX records).
  • Query the Quad 9 filtered recursive DNS service to see if a domain is blocked by Quad 9.
  • Using a commercial randomised IP proxy service, test each reported URI from 2 different geo located sources for their http and https status.
  • Query the Google Web Risk API to see if the URI has been flagged by google as an active threat.
  • Query the tranco list for the ranking of domains with identifiers.
  • If the TLD is RDAP enabled, connect with a credentialed user and look for create date, EPP status and other usefull data that may be redacted in WHOIS.
  • If zones are available, Arke can search for URL hijacking ( typosquatting ) activity.
  • Analyse the collected data and provide a daily report to ccTLD managers.
  • If the ccTLD registry system is RDAP enabled, Arke can send emails directly to domain contacts if there is evidence of a hosting compromise or an AUP violation.

Validation

DNS TLD and registrant nominated DNS
Google Safe Browsing API https://developers.google.com/safe-browsing/v4/lookup-api
BrightData Proxy Services https://brightdata.com/
Quad9 DNS Filtering Service https://www.quad9.net/

Changelog

[0.2.11] - 2021-11-26

  • Update URI compression sensitivity to handle more complex objects.
  • Resolved stick issue with transform component on unattended modes.
  • Updated category counts to reflect URI totals.
  • Changed category counter to only trigger on STATUS 200.
  • Updated null counter processor to ignore repeated uris.
  • Report text changes.

[0.2.10] - 2021-11-09

  • Created PLEXUS memory and thread management object.
  • Implemented the PLEXUS memory cloud into network processing pipe-line.
  • Implemented the PLEXUS memory cloud into extension processing pipe-line.
  • Created SBOOLEAN object for three-state binary.
  • Implemented three-state object into NXHOST variable.
  • Implemented three-state object into NXDOMAIN variable.
  • Report Changes

[0.2.9]

  • Resolved network freeze issue in panopticon.network namespace, set resolver time-out to 3seconds with 10 retries.
  • added cocca-nic source to source collection.
  • Updated wording in URI analysis table.
  • Added "ignore": [ "source-name" ] array to reports for allowing to ignoring of certain sources per TLD.
  • created shared bridge for bypass.json, filter.json, nsuffix.json, and rdap.json in respective load points.
  • added system.worker mode to worker process.

[0.2.8]

  • Updated rdns, dns, resolve and info query probes to use indicator element instead of passed resource.
  • Moved buffer system to query layer to compensate for already queried resources in the domain vector.
  • Added HostNX, Hostv4, Hostv6 and HostInf fields int data pipeline.
  • Moved Host functions to hostdns extenrsion.
  • Integrated the new host fields into records.csv.
  • Integrated the new host fields into records.json.
  • Added Active Host count to top domains list.
  • Resolved exception in transform component.
  • Removed domain tables as per Request.

[0.2.7]

  • Created this change log.
  • Updated archive and print procedures to include change-log.
  • Added topography counter to denote for external and registry originated domains.
  • Updated Domain Threat Table with topography values.
  • Created new unique-list document for domain names that are registered extrernal to the registry (external-[tld].asc).
  • Updated language for opening statement, changed the word `unique domains` to `unfiltered domains`.
  • Changed language from `Count` to `URI Count` in top reported domains.
  • Added pause capability per service node on generated reports. "pause": X, where is a positive integer in seconds.
  • Added Hex Tracer (http://tracker.h3x.eu/) data source.
  • Resolved out of memory issue when constructing records.json, implemented custom buffered json writer.


Sources

mw-patrol https://www.malwarepatrol.net
apwg-ecrimex https://apwg.org
tranco https://tranco-list.eu
phish-tank https://www.phishtank.com
phishing-army https://phishing.army
open-phish https://openphish.com
pihole-bl https://github.com/mhhakim/pihole-blocklist
badboyz-hosts https://github.com/mitchellkrogza/Badd-Boyz-Hosts
Hex Trace http://tracker.h3x.eu/
targeted-threats https://github.com/botherder/targetedthreats
oshosts-bl https://awesomeopensource.com/project/notracking/hosts-blocklists
vx-vault http://vxvault.net
ultimate-hbl https://github.com/Ultimate-Hosts-Blacklist/Ultimate.Hosts.Blacklist
url-haus https://urlhaus.abuse.ch
alpha-soc https://alphasoc.com
blocklist-project https://github.com/blocklistproject
firebog https://firebog.net
kad-hosts https://kadantiscam.netlify.app
digitalside-osint https://osint.digitalside.it
threatshub-org https://www.threatshub.org
m-krozga https://github.com/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites
viri-back https://tracker.viriback.com
malsilo-feed https://docs.fortinet.com/document/fortisoar/1.0.0/malsilo/75/malsilo-v1-0-0
dzone-bl https://github.com/oznu/dns-zone-blacklist
zfiles-dlist https://zonefiles.io/compromised-domain-list
lehigh-bl http://malwaredomains.lehigh.edu
joe-wein https://joewein.net
sb-unified https://github.com/StevenBlack/hosts