12 December 2016
Tweet from MeydanTV.
The 28th of November 2016, we received information from trusted sources[1] inside Azerbaijan that at least two independent news sites were loading poorly, namely Azadliq and Voice of America.
Although some web requests seemed to arrive to azadliq.info and no obvious full blocking was talking place, we concluded that the only possible reason that the website was not loading properly was a consequence of packets being discarded inside of Azerbaijan.
During the last two weeks we have conducted a set of tests to identify the root cause of the poor performance of the websites inside the country. This report summarizes some of our technical findings.
[1] Tweet from MeydanTV (a Berlin-based Azerbaijani non-profit media organization. Founded by dissident blogger and former political prisoner Emin Milli in 2013).
First blocking attempt
Monday, 28th November 13.35 PM UTC
As a hosting provider of Azadliq’s website, we are monitoring nearly hundred different parameters that can impact the performance of their website. We keep historical data of such indicators, which allow us to spot changes in latency, jitter or traffic behaviour.
Once we concluded that the website was not affected by a general routing or network outage, we looked closely into three parameters: (1) TCP retransmissions, (2) Duplicate ACK packets and (3) TCP Segments Lost. These three parameters alone gave us a good fingerprint of what looked like artificially engineered bandwidth throttling and network congestion.
Distribution of HTTP requests arriving to azadliq.info from Azerbaijani ASNs.
Table 1 shows the distribution of HTTP requests arriving to the Azadliq website from Azerbaijani ASNs that seem problematic.
After eight hours of testing, we decided to enforce HTTPS on all content of the website as traffic throttling was only affecting port 80 (HTTP). Moving the website to HTTPS, unblocked the artificially created congestion and the website was fully reachable from Azerbaijan.
Graph 1: The retransmissions dropped dramatically at 22PM when the site was moved to HTTPS.
Second blocking attempt
Wednesday, 7th December 15.15 PM UTC
Ten days after identifying the network congestion, we witnessed again TCP retransmissions. After a new round of tests, we concluded that the problem was caused by the same technique used the 28th of November 2016.
The graph shows the increase of TCP retransmissions and duplicate packets and how such retransmissions stop at soon as we move the website to another IP location.
Graph 2: Retransmissions reappeared on the 7th of December, but stopped when the site was moved to a new location.
Graph 4 shows congestion by looking into the performance indicators of TCP Reno, an algorithm used to handle congested links. The graph plots the “ACK Recoveries”, that helps us to match the time when the congestion start and to check if our countermeasures are effective.
Graph 3: TCP congestion avoidance algorithm (Reno) displaying ACK Recoveries.
Third blocking attempt
Saturday, 10th December 09.15 PM UTC
A new throttling is implemented the Saturday morning, our graphs shows immediately the increase of TCP retransmissions in the sever side.
Graph 4: The increase of TCP “Fast Retransmits” indicate according to RFC 2581 the arrival of 3 duplicate ACKs (4 identical ACKs without the arrival of any other intervening packets) as an indication that a segment has been lost.
According to the data showed in Graph 5, we can see that duplicate ACKs from inside Azerbaijan arrive to our server that indicate that traffic is dropped from the server to client direction (downstreams).
Graph 5: Two of the three blocking attempts (increase of segments retransmitted).
Forth Blocking Attempt
Thursday, 15th December 2016
Around 7:30 AM UTC, azadliq.info was fully unreachable from inside Azerbaijan until 12 PM (UTC), then congestion started again until we decided to move the website one more time at 15.00 PM UTC.
The following graphs shows the period where full block was effective against the site. 
Around 12 PM UTC, the heavy blocking was released and a new “artificial congestion” was in place. During this three hours period, we recorded the values of TCP retransmissions in one minute periods. This indicators are currently used to trigger alarms in our system so we can promptly detect new attempts.
After collecting enough information, we re-allocated the site for the forth time.
Fifth-Sixth Blocking Attempt
Saturday, 24th December 2016 (00:00 Baku time) 27th Anniversary of Azadliq Newspaper
The fifth attempt to block the site started on Friday, 23rd December 20 PM UTC.
The following providers were having issues reaching the site
34 AZ, Azerbaijan – AS28787 Baktelekom
26 AZ, Azerbaijan – AS34170 Azerbaijan Telecomunication ISP
20 AZ, Azerbaijan – AS31721 Azercell Telecom AS
16 AZ, Azerbaijan – AS29049 Delta Telecom LTD.
10 AZ, Azerbaijan – AS57293 AG Telecom LTD.
10 AZ, Azerbaijan – AS15723 Azeronline Information Services
9 AZ, Azerbaijan – AS41997 Stellford LLC
5 AZ, Azerbaijan – AS39280 Ultel LLC
4 AZ, Azerbaijan – AS59523 CNC.AZ MMC
4 AZ, Azerbaijan – AS50274 Alfanet LLC
4 AZ, Azerbaijan – AS44725 AZQTEL
4 AZ, Azerbaijan – AS29584 AZEDUNET LLC
4 AZ, Azerbaijan – AS197830 Baksell LTD LLC
3 AZ, Azerbaijan – AS39397 Az.StarNet LLC
3 AZ, Azerbaijan – AS13099 AzEuroTel J.V.
2 AZ, Azerbaijan – AS199987 AvirTel LLC
1 AZ, Azerbaijan – AS61304 Connect LLC
1 AZ, Azerbaijan – AS42779 Azerfon AS
1 AZ, Azerbaijan – AS34876 SMART SISTEMZ TECHNOLOJI MMM
1 AZ, Azerbaijan – AS198448 MEQA-TELEKOM LTD
1 AZ, Azerbaijan – AS196821 ICC of Ministry of ICT of Azerbaijan
During the 24th of December, two blocking attempts took place. The first one started the 23rd of December (20 PM UTC – 24 PM in Baku), the second attempt in the same day started 7.30 AM UTC, 11.30 AM in Baku)
The impact of congestion-blocking
We have also collected Google Analytics for the past days to see the impact of the blocking-congestion and to have a third party organization recording such effects.
The graph shows Google Analytics for the 14th and 15th of December (Baku local time). As in our own monitoring system, the impact of the full-blocking is noticeable between 11.40 – 16.00 PM (Baku local time), the congestion is in place between (16.00 PM and 18 PM).
The following graph marks more clearly the two different periods (blocked traffic 1-2), and congested traffic (2-3).
Summary of the re-allocation of the website.
The 28th of November 2016 at 11PM, we enforced HTTPS as we could verify that traffic congestion was only taken place in port 80.
The 8th of December 2016 at 17PM, after more than 24h monitoring the congestion, we moved the site to yet another location. Moving the site to a new location automatically brought the traffic back to normal behavior.
The 11th of December 2016 at 18PM, we moved the site once again to a new location. Moving the site to a new location automatically brought the traffic back to normal behavior.
The 15th of December 2016 at 15 PM, we moved the site one more time after seven hours of different congestion filters implemented in Azerbaijan. The first five hours resulted in a full block of the website
The 24th of December 2016 at 2 AM, we moved the site to a new location. The blocking started the 24th December at 00:00 Baku time.
More technical Details about the blocking are available here