4 October 2017

The following document provides more technical details of the mechanisms used to block the websites associated to the support of the referendum of 1-O in Catalunya.

• The first section (PART I) covers the technical aspects of Movistar blocking.
• The second part (PART II) of the document shows how Denial of Service Attacks were coordinated from the discussion board “Foro Coches” the referendum day 1-O

This is a live document that we keep updating while we review forensic evidence and reports from people on the ground.

PART I: Network Interference (13th September 2017 -)

The number of websites blocked remains uncertain and we can only confirm 70 websites, much less number that reported in the media (140).

Presence of DPI in Telefonica / Movistar

The most technically advance technology for Internet monitoring and inspection seems to be present in Telefonica / Movistar network.

One of the websites blocked is the domain gateway.ipfs.io. IPFS is a peer to peer filesystem that allows to serve web content. The URL that was mirroring the referendum.cat seized domain is:

https://gateway.ipfs.io/ipns/QmZxWEBJBVkGDGaKdYPQUXX4KC5TCWbvuR4iYZrTML8XCR

Due to the impossibility of blocking this “specific URL”, Telefonica / Movistar opted to block the whole domain gateway.ipfs.io

Two technical mechanisms are implemented in the network: (1) Blocking the HTTP requests by means of “Host” HTTP header inspection and (2) Blocking of the SSL connections by deep packet inspection of “SSL Hello Client” messages.

HTTP Inspection

The following packet trace shows how HTTP connections are handled by a specialized hardware. Notice the change of TTL values in the packets. Packets from the legitimate destination (gateway.ipfs.io) have a TTL value of 53 while injected packets have a TTL of 249.

The legitimate server sends packets with TTL=64 while the DPI gear sends packets with TTL=255. The legitimate server is 64-53=11 hops away, while the gear that injects the traffic is 255-249=6 hops away within Telefonica infrastructure.

In the case of HTTP connections a HTTP 403 response is sent back to the client. The HTTP response seems a template that is used to filter other websites. The variable PHISHING_TSOL_MENSAJE_1 might indicate that this template is used by the Antiphishing group of Telefonica Solutions or “Tsol”.

In this filtering template three different servers can be used to redirect the connections that are filtered:

 window.location.replace("http://195.235.52.40");
 window.location.replace("http://paginaintervenida.edgesuite.net");
 window.location.replace("http://webbloqueadaporpolicianacional.com");

In the case of the referendum related pages, all sites are redirected to paginaintervenida.edgesuite.net hosted at Akamai.

The template page indicates that different law enforcement authorities in Spain have different “parking locations” for filtered sites. The Guardia Civil uses the Akamai network to park their seized or filtered domains while the National Police uses the server webbloqueadaporpolicianacional.com hosted with Arsys. Illegal gambling websites seems to be redirected to 195.235.52.40 within Telefonica TDE Network.

During a period of time another filtering template was used that included the code:

 case "Judicial_Guardia_Civil":
 text = "Judicial_Guardia_Civil"
 window.location.replace("http://www.marca.com");
 break;

We ignore the logic behind this code, that suggest a redirection of the blocked websites to the sport newspaper marca.com

HTTPS Inspection

As in the case of HTTP inspection the TTL values obtained from a packet trace indicate that a specialized hardware device is in the inside Telefonica network.

In the case of SSL, filtering is taking place by injecting RST traffic after the Deep Packet Inspection gear detects a SSL Hello Client message with the domains to be filtered.

Delaying protocol handshakes to detect DPI presence

DPI is binded to specific destination IP addresses

Tests confirm that DPI inspection is only applied in all packets with certain IP destinations. For example, if the websites are hosted in a CDN like Cloudflare, the DPI will only inspect the IP addresses associated to the domain name in that network making possible to retrieve the content from other IP addresses in the CDN provider.

DPI keeps session state for 10 seconds

The DPI gear keeps session state for 10 seconds, delaying the HTTP protocol handshake allows to bypass the DPI filtering.

For example the following script establishes a connection (using netcat nc) to the blocked website “guardiacivil.sexy” and applies time delays to the HTTP GET request.

If we delay the GET request more than 10 seconds the DPI inspection is not longer tracking the TCP session.

function input {
sleep 11 
echo "GET / HTTP/1.1"
echo "Host: guardiacivil.sexy"
echo
echo
}

input | nc guardiacivil.sexy 80

Update 2017-10-03: Operators like Jazztel are redirecting the IP address: 82.223.11.66 that corresponds to a website in Arsys with the message:

PART 2: REFERENDUM DAY 1-O

Was bulk DPI inspection was used to identify the “Central Count Servers”?

During the opening of the polls at 9 AM the 1-O, the “Central Count Server” hosted in Cloudflare was blocked and Internet connection was shutdown in several voting stations.

During the morning of the 1st of October several alternative IP (proxy) addresses were blocked. The celerity of how this blocking were implemented make us believe that DPI technology was also used to monitor connections coming from voting stations to identify and sabotage the IT infrastructure during voting.

Other people reporting from the voting stations claim that infiltration was also a feasible option to leak the IP addresses of the proxies.

Denial of Service against registremeses.com

At 9 AM, the 1st of October, the website registremeses.com that was responsible to validate the ID cards of the voters was no longer reachable.

The website was registered the 29th of September, two days before the Referendum and hosted behind Cloudflare from the early morning of the 1st of October (alina and sid DNS servers).

Did Foro Coches coordinate a Denial of Service attack the 1-O?

Information gathered from Google Searches and cache versions of the discussion forum indicate that Denial of service attacks were likely coordinated from (at least) the Forum “Foro Coches”, where the domain and the alternative IP addresses were published.

The link to the discussion thread is https://www.forocoches.com/foro/showthread.php?t=5933224 but now has been removed by the creator “alextango”

Alextango claimed in the Forum “Os recuerdo que hacer ddos contra algo que de facto es ilegal, no es ilegal!” I want to remind you that to DDOS something that is ilegal, it is not ilegal!.

One of the posts in the thread:

There is also messages in the Forum that indicate that the servers were only reachable from the Catalan IP space and the attackers  were aware of that:

207.154.210.53, 104.27.130.101 y  104.27.131.101 Sólo son accesibles desde direcciones IP catalanas, pero si se hacen las suficientes

Reports of the blocking were also available via social media

Some twitter accounts were created to announce the IPs

Did the DDOS attack take place?

In order the find out if the attack took place we have collected the IP addresses that we could find from the servers used for the validation of voters.

From public sources we also obtained the IP 34.241.22.106 as showed in the picture.

We looked into our “darknet” sensors that work as described in this document from caida.org and we obtained a relevant sample:

08:40:32.490444 IP (tos 0x0, ttl 44, id 0, offset 0, flags [DF], proto TCP (6), length 44)
34.241.22.106.80 > X.X.X.124.31365: Flags [S.], cksum 0xcdfd (correct), seq 4101521145, ack 1295182152, win 26883, options [mss 1360], length 0

Users in the Foro Coche Forum will also posting when the sites were going down (TANGO DOWN!)

What does this recorded data show us?

The recorded data in our darknet indicates that a SYN flooding attack using “spoofed” IP addresses was performed against the server at 08:40 UTC (10:40 AM). This data indicates that the attackers did not only asked people to reload the websites with traffic but used a “Denial of Service” infrastructure to perform the attacks.

The evidence collected strongly suggests that DDOS attacks took place during the event.

Update: 4-OCT-2017

• Received several screenshots from the discussion Forum  where the attacks were coordinated

Update: 5-OCT-2017

• alextango  has just removed his account from Foro Coches (user 29792 Register: 19-may-2004)

• alextango has just removed his personal page from Blogger

Forensic Evidence

Due to the public interest of this case,  we are sharing the following forensic evidence to help others with this investigation.

All the information has been obtained from public sources.

• 77 screen shoots from the Foro Coches (alextango ddos #novotaran)  [GALLERY]
• Maltego Graph that summarizes our attribution analysis [DOWNLOAD]

• A internal document that summarizes the steps we followed to track down the coordinator of the referendum 1-O DDOS attacks [TXT]
• The attacker has already removed most of the public tracks but most of the information can be retrieved from Search Engine Cache or alternatively from the following archive.is pages.

Finally, we want to thank all who reached out to help with this research. Thanks for all your hints and information from the ground.