How operators use Sandvine to block independent media in Egypt

21 September 2020

Internet blocking is a common tool for the Egyptian authorities to silent dissidents or those that think and speak freely. Currently, more than 600 websites are blocked, more than hundred of those are media and news websites.

In April 2020, Internet operators in Egypt made changes to their blocking strategy and started to block subdomains by wildcard. This change resulted in that many blocked Egyptian media, that used rotation of subdomains as a way to cope with the blocking, were once again unreachable to the Egyptian readers.

One of the blocked media that became affected of the new blocking method was Al Manassa, a popular citizen journalism platform. Al Manassa attracts young talents in the world of journalism and support them with a network of professional editors and a space to publish their writings. Al Manassa followed the rule book by the line when establishing the news outlet in 2016. They registered their website with the Information Technology Industry Development Agency and applied for a news website license at the Supreme Council for Media Regulation in 2018. Despite their attempts to comply with the publishing rules, their application for license was never approved (nor denied), and their website was blocked in June 2017 without any notification nor clarification.

Qurium believes that access to information is a key element for critical thinking and public education. Widespread blocking of information, without logic or reasoning,  is not the way to protect a population. “The government of Egypt is depriving its population from information, which is the reason we support our hosted organizations with technical and forensic analysis capabilities”, says Ester Eriksson, Managing director at Qurium.

Al Manassa has survived with the blocking by rotating subdomains using its original domain almanassa.com. Although this trick has helped with the blocking in the past, both the technical team and the readers have paid a high price with ever changing URLs as the subdomains quickly got blocked by operators. In April 2020, all subdomains under almanassa.com were suddenly blocked. The operators had applied a wildcard filter on the subdomains and all of them became unreachable. Al Manassa responded by changing TLD to almanassa.run, but the new domain was shortly blocked as well.

How is blocking implemented?

To better understand the Internet blocking in Egypt, Qurium has together with Al Manassa investigated the blocking methods in place and fingerprinted the Deep Packet Technology (DPI) deployed inside of the three operators Telecom Egypt (state owned), Orange Egypt (formerly known as MobiNil) and Vodafone.

Telecom Egypt (TE) and Orange Egypt

Qurium’s tests reveal that both TE and Orange Egypt are implementing blocking by means of Sandvine hardware. 

[SYN] IP (tos 0x0, ttl 64, id 21959, offset 0, flags [DF], proto TCP (6), length 60)     192.168.1.105.40114 > 104.18.33.66.80: Flags [S]1460,sackOK,TS val 1

[SYN-ACK] IP (tos 0x48, ttl 52, id 0, offset 0, flags [DF], proto TCP (6), length 52) 104.18.33.66.80 > 192.168.1.105.40114: Flags [S.]

[ACK] IP  (tos 0x0, ttl 64, id 21960, offset 0, flags [DF], proto TCP (6), length 40) 192.168.1.105.40114 > 104.18.33.66.80: Flags [.]

== Web request
[HTTP GET] IP (tos 0x0, ttl 64, id 21961, offset 0, flags [DF], proto TCP (6), length 117)      192.168.1.105.40114 > 104.18.33.66.80: Flags [P.]

     GET / HTTP/1.1
     Host: ALmanaSSa.com
     Accept: /

[Sandvine] IP (tos 0x0, ttl 59, id 13330, offset 0, flags [none], proto TCP (6), length 40) 104.18.33.66.80 > 192.168.1.105.40114: Flags [R.]

After completing the 3-way handshake, once a request is sent to the domain “ALmanaSSa.com”, a Reset Packet arrives to tear down the connection. The Reset Packet has two signatures:

  • TTL=59: The Time to Live of the packet is 59 instead of 54 that suggest that the Reset packet comes from a device closer to the user that the real website.
  • ID 13330: The IPID field of the packet is 13330 that suggest the presence of a Sandvine/Packetlogic device in Mobinil/Orange. Same hardware that Qurium previously detected in Jordan.

Transparency Proxy used for HTTP requests

HTTP requests in Orange return the HTTP Error 502 with a webpage with the comment <!– default “Server Hangout” response (502) –>

This error message is in the code of Yahoo Traffic Server, now known as Apache Traffic Server.

The presence of a “full transparent proxy” including the outgoing transparency that hides the proxy from the hosting provider is consistent with the following facts:

  • ttl: TTL values of the SYN-ACK and ACK packets coming to the client are not consistent with traffic coming from the hosting server.
  • mss: The maximum segment size is also modified, suggesting the presence of a IP tunnel
  • syn-ack immutable values: win 63443, wscale 6
  • sequence number translation: seq numbers are translated in transit.

The forensic evidence strongly suggest that Apache Traffic Server is being used to block HTTP traffic, most probably diverting the traffic using Cisco ASA and WCCP protocol.

Which protocols are being monitored?

Once we discovered the presence of the “Transparency Proxy” setup in Orange we proceed to determine which port numbers are intercepted and what IP ranges were monitored.

To our surprise we discovered that all routable IP addresses in ports 80, 443, 25 and 8080 are monitored. 

Starting Nmap 7.60 ( https://nmap.org )
 Nmap scan report for 172.16.20.11
 Host is up (0.036s latency).
 Not shown: 996 filtered ports
 PORT     STATE SERVICE
 25/tcp   open  smtp
 80/tcp   open  http
 443/tcp  open  https
 8080/tcp open  http-proxy

Two interesting findings of the research is that outgoing Mail in port 25 is being proxied and private IP space in the Internet is being tracked.

The table below presents a summary of the blocking methods used by the three operators.

ProviderTEOrangeVodafone
HTTPSandvineATS – WCCP
Full Transparency		Allot?
HTTP-SIGRST Injection
TTL 59
0x3412
MSS 1432 (GRE?)		HTTP 502
MSS 1380 (Cisco ASA?)
SYN trans		RST Inject
TTL 250
MSS 1300
HTTPS
Sandvine
Sandvine
HTTPS-SIGSYN Proxy
RST
ID 13330
0x3412		SYN Proxy
RST
ID 13330
0x3412		RST Inject
TTL 250
MSS 1300

Sandvine in Egypt

The presence of Sandvine in the region is not new. Several press releases of the company announced partnerships with different operators. Sandvine is known to sell their technology for Internet blocking worldwide. Both Qurium and CitizenLab have previously reported that Sandvine provides Internet blocking capabilities in countries like Azerbaijan, Jordan, Turkey and Egypt.

Sandvine DPI and traffic impact in our infrastructure

When Sandvine is used to block websites hosted with Qurium, it has implications on how traffic reaches our servers. The impact of their traffic tampering is not logged by default in webservers as the web sessions are terminated before a single web request arrives.

In order to better understand the techniques used to block the websites, Qurium has deployed HAProxy and logged all SSL related errors.

With the logs and the analysis of packet captures, we can conclude that three reported errors are related to Sandvine DPI hardware.

  • Connection error during SSL handshake: Reset packets coming from Sandvine. RST-ACK IPID Sandvine after 3WHS.
  • SSL handshake failure: Silent discard of the flow after HTTPS Hello Client.
  • Timeout during SSL handshake: Silent discard after 3WHS.

Due to the architecture of Sandvine, that performs flow-based analysis in multi-core hardware, we have seen how a single web-request by a reader can trigger different SSL errors associated to each of the individual flows created by the browser.

For TE-Data, we have estimated that 3/4 of the connections/flows are teared down by Sandvine. Hence, 25% of all connections to blocked websites are going through the DPI from Sandvine.

Conclusions

  • Sandvine is being used to block access to independent media and human rights organizations in at least two providers in the country, state-owned Telecom Egypt and privately owned Orange Egypt (formerly known as MobiNil).
  • Sandvine DPI tears down the majority of connections established to blocked websites, but no less than 25% of the traffic goes through the DPI equipment.
  • The following parameters can fingerprint the existence of Sandvine equipment:
    • TTL: The Time to Live value is higher than expected which suggest that the Reset packet comes from a device closer to the user that the real website.
    • IPID: 13330

Media coverage