Offline Downloading in China: A Comparative Study

Размер: px
Започни от страница:

Download "Offline Downloading in China: A Comparative Study"

Препис

1 ACM SIGCOMM ACM SIGMETRICS Offline Downloading in China: A Comparative Study Zhenhua Li Christo Wilson Tianyin Xu Yao Liu Zhen Lu Yinlong Wang lizhenhua1983@gmail.com Oct. 30,

2 Outline Background Problem System Workload Performance Optimization 2

3 Internet Access across the World Internet Penetration Not only penetration, but also quality of Broadband: > 25 Mbps of Download Bandwidth Broadband: 4 10 Mbps, Unstable and Limited 3

4 Pains of the Developing World Downloading large files requires high-quality network connections! DTN - Delay Tolerant Networking? 4

5 The Case of Modern China Promises ü 46% of China s population has come online ü World-class companies like Tencent, Baidu, Alibaba, and Sina Weibo (Microblog) 5

6 The Case of Modern China Challenges n Over 72% of China s Internet users have low-quality network connections Low access bandwidth Unstable/ unreliable connec5on ISP barrier (Poor inter- connec5vity between ISPs) Other reasons 6

7 Offline Downloading in China Internet Time 2. predownload start pre-downloading Data flow finish pre-downloading Proxy 1. request 3. fetch file request An HTTP/FTP/ P2P link start fetching User free to be offline finish fetching 7

8 Typical Implementation (1): Cloud-based Internet 2. predownload Proxy 1. request 3. fetch User Caches PBs of files in a datacenter that is within or directly peered with the requesting user s ISP Tencent Xuanfeng Xunlei Baidu CloudDisk 8

9 Typical Implementation (2): Smart AP-based Internet 2. predownload Proxy 1. request 3. fetch User Caches data in an embedded or connected storage device, e.g., an SD card, a flash drive, or a disk drive HiWiFi MiWiFi Newifi 9

10 Great Success in Industry Tencent Xuanfeng ü Over 30M users HiWiFi > 1.5M shipments Xunlei ü Over 80M users MiWiFi > 2M shipments Baidu CloudDisk ü Over 150M users Newifi > 0.6M shipments 10

11 Problem 11

12 The 1st Problem Is offline downloading really effective in most cases? 1. request Internet 2. predownload Proxy 3. fetch User High success rate? High speed? 12

13 The 2nd Problem Which offline downloading approach should be selected? Both? 13

14 The 3rd Problem When is offline downloading useless or even worse? 1. request Internet 2. predownload Proxy 3. fetch User User- side access bandwidth Cloud- side service capability ISP barrier Transfer protocol File popularity Hardware & filesystem 14

15 General Problem: Selection Dilemma Common Downloading or Offline Downloading? Cloud-based or Smart AP? And which smart AP? Our work is the first quantitative and comparative study on these problems based on a large-scale dataset from Xuanfeng cloud and benchmark experiments of popular smart APs. 15

16 System 16

17 Xuanfeng Cloud Internet requests DB files Pre-downloading servers 2-PB Storage Collaborative servers Cache Uploading servers ISPs Privileged Network Path 17

18 Smart APs $20 Opkg applications Shell OpenWrt operating system $100 Internet NIC (xdsl) CPU RAM NIC (WiFi) $20 SATA Interface USB Interface SD Interface User Device 18

19 Workload 19

20 Xuanfeng Dataset q Complete running logs during a whole week in 2015, involving 4M tasks, 0.78M users & 0.56M unique files User Requests User ID IP address Access bandwidth Request Ime File type File size Original data source Transfer protocol Pre- downloading Trace Start Ime Finish Ime Acquired file size Traffic usage Cloud cache hit Avg. speed Peak speed Success or failure Fetching Trace User ID IP address Access bandwidth Start Ime Finish/pause Ime Acquired file size Traffic usage Avg. speed Peak speed 20

21 File Type, Size & Transfer Protocol File Type 10% 15% 75% Video SoYware Other Transfer Protocol [ 值 ] [ 值 ] [ 值 ] BitTorrent emule HTTP/FTP Median: 115 MB Average: 350 MB Maximum: 4 GB 25% files < 8 MB 21

22 File Popularity Zipf Power law SE Stretched Exponential Matthew effect (for non-videos) + Fetch-at-most-once effect (for videos) 22

23 Smart APs: Benchmark ADSL Link Sampled workload from the Tencent Xuanfeng dataset HiWiFi Performance data Internet MiWiFi Newifi Storage server *Note: We assume that the smart AP based offline downloading systems have similar workload characteristics to Xuanfeng, since most end users are not familiar with the technical details and cannot differentiate these services. 23

24 Performance 24

25 Xuanfeng: Pre-downloading Speed and Fetching Speed Median: 25 KBps Average: 69 KBps Median: 287 KBps Average: 504 KBps Owing to the privileged network path, Xuanfeng significantly improves users perceived downloading speeds by 7 11 times (fetching speed / pre-downloading speed) 25

26 Xuanfeng: Unsatisfactory Fetching Speed 1 28% of fetching speeds are below 125 KBps (= 1 Mbps, typical playback bitrate of HD videos) 9.6% ISP barrier 10.8% Low user-side access bandwidth 1.5% Lack of cloud-side upload bandwidth 6.1% Unknown... The cloud-based approach performs poorly once there is a bandwidth bottleneck in the privileged network path between the cloud and the user 26

27 Xuanfeng: Shortage of Cloud Bandwidth 1.5% Lack of cloud-side upload bandwidth u 0.84% of highly popular files account for 39% of all downloads u 87% of requested files are hosted in peer- to- peer (P2P) data swarms 2 The cloud is threatened by running out of upload bandwidth due to unnecessarily sending highly popular P2P files. As the user base continues to grow, the cloud will have to reject more (>1.5%) fetching requests. 27

28 Xuanfeng: Pre-downloading Failure requests DB files X 2-PB Collaborative Caching à 8.7% Failure à 16.4% Failure The cloud cache effectively avoids nearly half of predownloading failures High popularity Low failure ratio 28

29 Smart APs: Pre-downloading Failure Failure Ra5o Xuanfeng Cloud Smart APs Overall 8.7% 16.8% Unpopular files 13% 42% 86% Insufficient seeds in a peer swarm 10% Poor HTTP/ FTP connections u 36% of offline downloading requests are issued for unpopular files 4% Unknown... 3 Smart APs frequently fail during pre-downloading unpopular files 29

30 Smart APs: Pre-downloading Speed Speed Xuanfeng Cloud Smart APs Median 25 KBps < 27 KBps Average 69 KBps > 64 KBps 4 A smart AP s pre-downloading speed can be restricted by its hardware and/or filesystem, since some types of storage devices and filesystems do not fit the pattern of frequent, small data writes during pre-downloading 30

31 Smart APs: Pre-downloading Speed NTFS is incompatible with the OpenWrt OS USB flash drive is unsuitable for frequent, small data writes 31

32 Performance Summary Xuanfeng Cloud Bottleneck 1: Unsatisfactory fetching speed Bottleneck 2: Shortage of cloud bandwidth Merit 1: Effective avoidance of predownloading failures Merit 2: No hardware cost at the user side Smart APs Merit 3: Stably high fetching speed Merit 4: No cloud infrastructure Bottleneck 3: Frequent failures during pre-downloading Bottleneck 4: Hardware/filesystem restrictions on pre-downloading The two approaches are subject to distinct performance bottlenecks L while also being complementary to 32

33 Optimization 33

34 ODR Middleware q Help users automatically select a proper (offline) downloading way ODR (Offline Downloading Redirector) 2. Query DB Xuanfeng 1. Request 3. Redirect Cloud User device Smart AP Storage device p Primary goal: minimizing the downloading time and failure ratio p Secondary goal: minimizing the upload bandwidth burden on the cloud 34

35 ODR Web Site 35

36 ODR Design & Performance P2P file? Y N Data source: Original Bottleneck 4? Y N User device Smart AP Heuristic rules extracted from measurement findings Bottleneck 2 Highly popular file? Bottleneck 3 Y N HTTP/FTP file Bottleneck 1? Y N Cloud + Smart AP Cloud Data source: Cloud Cloud cache hit? Y N Cloud pre-download Cloud predownload succeed? Y N fail Even using this coarsegrained design, we manage to remarkably reduce the current performance bottlenecks of offline downloading! 36

37 Summary p What Offline Downloading is and Why it is popular in China p Selection Dilemma of Offline Downloading p Measurement findings of Xuanfeng and popular smart APs The two approaches are subject to distinct performance bottlenecks, while also being complementary to each other. We feel that offline downloading has broad applicability to other areas of the world that lack broadband penetration. By deploying offline downloading technologies, coupled with our proposed ODR middleware, the Internet experiences for users in many parts of the world can be improved.

38 Thank you! Any ques5ons? Xuanfeng is pronounced as [ʃuæn fəŋ] or simply XF HiWiFi Xunlei is pronounced as [ʃunlei] or simply XL MiWiFi Baidu CloudDisk Newifi

39 Backup slides

40 Xunlei, Xuanfeng, and Baidu CloudDisk Xunlei Ø Launched in 2009 ü Over 80M users Tencent Xuanfeng Ø Launched in 2010 ü Over 30M users Baidu CloudDisk Ø Launched in 2012 ü Over 150M users

41 Xunlei, Xuanfeng, and Baidu CloudDisk Xunlei ü Over 80M users Tencent Xuanfeng ü Over 30M users p Charges each user nearly $2 per month p Conditionally free --- should be the VIP user of Tencent Baidu CloudDisk ü Over 150M users p Totally free --- the user even does not need to register

42 Offline Downloading outside China u Besides those developing countries, developed countries can also benefit from offline downloading (based) services. SMART WI-FI ROUTERS USB storage interface

43 State-of-Art Downloading Techniques p CDN typically only help to deliver files for content providers who pay for the service. p The business model of offline downloading is the opposite of CDN, because it charges (or sometimes frees) its users, i.e., content receivers, for better downloading experiences. Offline Downloading DTN + reverse-cdn 1) in-network storage for caching 2) decoupling content senders and receivers 3) disruption tolerance 4) multi-party communication through replication 5) mobility and multi-homing

44 Performance: Delay Median: 7 minutes Average: 27 minutes Median: 82 minutes Average: 370 minutes Xuanfeng reduces users perceived downloading delay by times

45 Performance: End-to-End Median: 233 KBps Average: 380 KBps The collaborative cache of Xuanfeng remarkably avoids 89% pre-downloads Median: 10 minutes Average: 68 minutes

GPI Document

GPI Document Позиция Кол. Описание 1 DUOLIFT.27.AP3B..8.3 Продуктов номер: 9917762 Забележка! Показаната снимка може да се различава от реалната. Wastewater collecting tank with a total volume of 27 liter. Incl. pipeset

Подробно

Правилник на кампанията Ловци на оферти В периода 5 8 март 2019г. Данте интернешънъл С.А. (Dante International S.A.) самостоятелно, както и някои от м

Правилник на кампанията Ловци на оферти В периода 5 8 март 2019г. Данте интернешънъл С.А. (Dante International S.A.) самостоятелно, както и някои от м Правилник на кампанията Ловци на оферти В периода 5 8 март 2019г. Данте интернешънъл С.А. (Dante International S.A.) самостоятелно, както и някои от маркетплейс търговците на emag.bg, всеки търговец поотделно,

Подробно

PowerPoint Presentation

PowerPoint Presentation Събития, организирани в рамките на Scientix, през 2018 в България Евгения Сендова Институт по математика и информатика при Българска академия на науките National Seminar in Education with Scientix workshop:

Подробно

GPI Document

GPI Document Позиция Кол. Описание 1 UNOLIFT.27.AP3B..6.A Продуктов номер: 99144937 Забележка! Показаната снимка може да се различава от реалната. Wastewater collecting tank with a total volume of 27 liter. Incl. pipeset

Подробно

Microsoft PowerPoint - Windows2003 RRAS.ppt

Microsoft PowerPoint - Windows2003 RRAS.ppt Windows2003 Remote Access Христо Вълчанов hristo@tu-varna.acad.bg Remote Access конекции LAN Remote Access Protocols LAN Protocols Remote Access Protocols Internet LAN Protocols Remote Access сървър Remote

Подробно

УПЪТВАНЕ Facebook Телефон: Whatsapp: PowerLocus Истинските Безжични Bluetooth С

УПЪТВАНЕ Facebook   Телефон: Whatsapp: PowerLocus Истинските Безжични Bluetooth С УПЪТВАНЕ Facebook : @PowerLocus Email : powerdirect155@gmail.com Телефон:0885909483 Whatsapp: +31 633242189 PowerLocus Истинските Безжични Bluetooth Слушалки Как да включите PowerLocus? Моля, натиснете

Подробно

GPI Document

GPI Document Позиция Кол. Описание UPA5-20 AUTO Продуктов номер: 98699677 Забележка! Показаната снимка може да се различава от реалната. Applications Domestic applications requiring high pressure in taps, showers,

Подробно

NOVETHERM ВХОДНИ ВРАТИ

NOVETHERM ВХОДНИ ВРАТИ ВХОДНИ ВРАТИ TОВА, КОЕТО НИ ОТЛИЧАВА? WHAT DISTINGUISHES US? Детайлите са нещо, на което ние винаги обръщаме внимание! Алуминиеви входни врати за дома с най-високо качество, произведени специално за Вас.

Подробно

GPI Document

GPI Document Позиция Кол. Описание 1 UPA 15-90 160 Продуктов номер: 59539521 Забележка! Показаната снимка може да се различава от реалната. Applications Domestic applications requiring high pressure in taps, showers,

Подробно

ПОЛИТИКА НА КангаКодърс ЗА ЗАЩИТА НА ЛИЧНИТЕ ДАННИ СЪДЪРЖАНИЕ: 1. Увод 2. Кой е администратор на личните данни, събирани в този сайт? 3. Какви лични д

ПОЛИТИКА НА КангаКодърс ЗА ЗАЩИТА НА ЛИЧНИТЕ ДАННИ СЪДЪРЖАНИЕ: 1. Увод 2. Кой е администратор на личните данни, събирани в този сайт? 3. Какви лични д ПОЛИТИКА НА КангаКодърс ЗА ЗАЩИТА НА ЛИЧНИТЕ ДАННИ СЪДЪРЖАНИЕ: 1. Увод 2. Кой е администратор на личните данни, събирани в този сайт? 3. Какви лични данни КангаКодърс събира в този сайт? 4. За какви цели

Подробно

PowerPoint Presentation

PowerPoint Presentation TOWEL RADIATORS AND HEATERS ЛИРИ ЗА БАНЯ И НАГРЕВАТЕЛИ АЛУМИНИЕВИ ЛИРИ СТОМАНЕНИ ЛИРИ ЕЛЕКТРИЧЕСКИ ЛИРИ НАГРЕВАТЕЛИ ЗА ЛИРИ ALUMINIUM TOWELS STEEL TOWELS ELECTRICAL TOWELS ELECTRICAL HEATERS FOR TOWELS

Подробно

GPI Document

GPI Document Дата: Кол. UPA 5-90.9.209 г. Забележка! Показаната снимка може да се различава от реалната. Продуктов номер: 99538895 Grundfos UPA 5-90 is a domestic booster designed for pressure boosting of drinking

Подробно

PowerPoint Presentation

PowerPoint Presentation Промени в тарифата за индивидуални клиенти от 18 септември 2019 г. CHANGES IN TARIFF FOR INDIVIDUAL CLIENTS AS OF SEP 18 th, 2019 Касови операции Cash operations Теглене в брой на каса до 2000 лв. или

Подробно

17

17 ЩЕ ИЗЧЕЗНЕ ЛИ БЪЛГАРСКАТА НАЦИЯ ПРЕЗ XXI ВЕК? Доц. д-р Стефан Стефанов катедра "Математика и статистика" СА "Д. А. Ценов" - Свищов (Продължение от брой 4/2003) Резюме: В работата се разглеждат измененията

Подробно

PL_LT_ALTHERMA_R32_HEATING_2019.numbers

PL_LT_ALTHERMA_R32_HEATING_2019.numbers монофазен идробокс The technology combines 2 EHBH04D6V EHVH04S18D6V EHVH04S23D6V EHBH08D6V EHVH08S18D6V EHVH08S23D6V EHBH08D6V EHVH08S18D6V EHVH08S23D6V HO 4 590,00 6 685,00 6 820,00 4 830,00 6 927,50

Подробно

Microsoft PowerPoint - Windows2003 Intro.ppt

Microsoft PowerPoint - Windows2003 Intro.ppt Windows2003 Server Въведение Надежда Рускова ruskova@tu-varna.acad.bg Историческо развитие Развитие и влияние на Windows server 2003 VMS VMS OS/2 OS/2 Влияния Netware Netware UNIX UNIX TCP/IP TCP/IP and

Подробно

Стандарти на бранда Brand Guidelines

Стандарти на бранда Brand Guidelines Стандарти на бранда Brand Guidelines Съдържание Contents Кои сме ние 4 Who we are 4 Как изглеждаме 7 How do we look 7 Елементи на марката 8 Brand elements 8 Лого Лого използване Лого използване: Най-голям

Подробно

14

14 ЩЕ ИЗЧЕЗНЕ ЛИ БЪЛГАРСКАТА НАЦИЯ ПРЕЗ XXI ВЕК? Доц. д-р Стефан Стефанов катедра "Математика и статистика" СА "Д. А. Ценов" - Свищов (Продължение от брой 1/003) Резюме: В работата се разглеждат измененията

Подробно

Доставка на компютърно и мултимедийно оборудване за "Факултет по обществено здраве, здравни грижи и медицина" при Университет "Проф. д-р Асен Златаров

Доставка на компютърно и мултимедийно оборудване за Факултет по обществено здраве, здравни грижи и медицина при Университет Проф. д-р Асен Златаров ТЕХНИЧЕСКА СПЕЦИФИКАЦИЯ ц бща ц бща ц без ДДС I ПРЕНОСИМИ КОМПЮТРИ 1 1 ПРЕНОСИМ КОМПЮТЪР 99 1500 148500 123750 Прцеср Intel Core i5 или еквивалент Екран 15.6" HD AntiGlare LED 8GB 1600MHz DDR3 SSD 120

Подробно

Структура на програма в C - Част 6 - goto, switch, break и continue клаузи

Структура на програма в C - Част 6 - goto, switch, break и continue клаузи Структура на програма в C Част 6 - goto, switch, break и continue клаузи Иван Георгиев, Христо Иванов, Христо Стефанов Технологично училище "Електронни системи", Технически университет, София 21 април

Подробно

New Generations 2

New Generations 2 Работата с младото покление минало, настояще и бъдеще ВЕСЕЛИН ДИМИТРОВ ПРЕДСЕДАТЕЛ КОМИТЕТ ЗА МЛАДИТЕ ПОКОЛЕНИЯ Комитет за Младите Поколения Ротари Ротари Спонсорство на Ротари ове със създанетите от тях

Подробно

r_AcademicCurriculum_BG

r_AcademicCurriculum_BG Утвърден от Академичния съвет на УАСГ Confirmed by Academic Council of UACEG Дата 13.07.2016 г. Date Утвърждавам Ректор Confirmed by Rector... / проф. д-р инж. Иван Марков / Prof. Dr. Eng. Ivan Markov

Подробно