DebTorrent
| Language: | Python |
|---|---|
| Debian bug: | bug - mixed up stats #487829 |
| Debian bug: | wishlist - HTTP download stats #488415 |
| Feature: | Graphics (as shown above) #515771 |
| Ongoing: | Documentation |
Debian is a non-profit Linux distribution, and relies on volunteers to provide HTTP and FTP mirrors. DebTorrent is one of the projects for end-users to contribute back to the system, by mirroring files that they've downloaded.
Both Cameron (the main Debian developer for it) and I are waiting to see if it gets more users; we're expecting it to get more use when the huge package archive comes online. It's currently functional, but not getting developer time. We both have todo lists that we'll work on if the usage picks up.
Projects with DebTorrent
After having a couple of small changes committed in time for Debian 5.0, I have a couple of development ideas running which I plan to contribute.
Showing which files are available from which peers
DebTorrent has a web interface to check the status of your peer. Graphically showing which files are available from which peers is both interesting and provides an insight in to the network behaviour. An approximation of the output can be seen above, and on the Debian Wiki.
(July 2008) My first prototype generated the images as SVG, inline in the status page's XHTML; but this made the status page slow to load. The code shows my learning curve of DebTorrent's class relationships. Cameron (DebTorrent's lead developer) was planning to have separate "quick" and "detailed" status pages, and I postponed finishing my code until that framework was available.
(February 2009) Both Cameron and I have had other projects to concentrate on instead of DebTorrent. Wanting to tidy up my contributions, I added framework for the quick and detailed status pages.
(September 2009) It's released in revision 415.
The graphics output is currently in SVG (easy to generate, and easy to include inline within XHTML). A future plan is to add PNG images using the pyimage library. One feature that SVG provides over a raster image is the ability to add tooltips showing which piece the mouse is over, and this will be lost once it moves to PNG.
Documentation
The learning curve for the design inherited from BitTorrent is quite steep. Now that I've gained some familiarity, I can contribute things such as a simple diagram of the main classes.
Optimisation
Another possible project would be to optimise the threading. The program is single-threaded, simulating threading via a loop that calls listener-like functions in between polls for I/O. Several parts of the program register polling callbacks when it's clear that the callback will merely schedule another poll.
Number of DebTorrent users
These statistics are from March 2009.
DebTorrent currently has around 200 users, and that's far below critical mass when those users are spread over at least five distributions (both Debian and Ubuntu) and many architectures. It is reducing bandwidth from the Debian mirrors by around 30% for those users, but this is minor given the number of users. A total of 0.3 TiB has been downloaded through DebTorrent, compared to the 200TiB of installation media which has been BitTorrented; I do not have stats for direct downloads.
I'm curious to find out what the trigger for people starting to use DebTorrent will be. The torrent part is dependent on the other users upload capacity; due to slow peers it always uses HTTP in parallel, even for some of the pieces that could be downloaded from a peer. Currently HTTP is always faster.
Huge data packages
Debian is currently rejecting software that depends on huge collections of data; both scientific research data and games. The amount of disk space required as much of a concern as the amount of bandwidth used, particularly as Debian relies on voluntary mirrors around the world, a huge package with few users will have an unwelcome footprint. The mirrors need disks that aren't cheap, in order to handle the wear and tear of serving normal packages.
Debian are now looking at setting up one server with the diskspace; which seems to be a situation that DebTorrent would compliment by handling the bandwidth.
Skills development
Python was a new language for me, and this gave me a working project to study.