Internet Mysteries: How Much File Sharing Traffic Travels the Net? -- Update
How much of the traffic on the internet is peer-to-peer file trading?
Everyone seems to agree it represents a lot of the traffic, but the truth is no one knows (with the possible exception of the ISPs and backbone providers in the middle, and they aren't telling or sharing raw data).
One of the most recent reports on P2P traffic came from a traffic optimization firm called Ellacoya in June 2007. Their report said that http-based web traffic had overtaken peer-to-peer traffic on the net, thanks to streaming media sites like YouTube.
Ellacoya, since acquired by Arbor Networks for its traffic-shaping technology, pegged http traffic at 46 percent of the net's volume, with P2P traffic close by at 37 percent.The company says the data was based on about 1 million North American broadband subscribers.
But little is known about when, how and where the company collected the data, or how it analyzed the packets.
Independent internet researchers, including KC Claffy of the Cooperative Association for Internet Data Analysis, ran their own tests in 2003 and 2004 -- following conflicting reports that file sharing was decreasing and increasing.
Using data from an internet backbone link in San Jose, California, the researchers found that P2P traffic was steady, if not increasing. For instance, BitTorrent grew some 100 percent in popularity from 2003 to 2004, but the researchers found that it was getting harder to track P2P bits, since P2P traffic was increasingly using encryption and random ports, making it harder to quickly identify the application that a packet was coming from.
The last time Sprint published an analysis of 30 large internet links (January 10, 2005), it found that file sharing accounted for less than 6 percent of the packets in the tube, with regular web traffic clocking in at more than 50 percent of the flow.
Speaking at Supernova conference last July, Claffy expressed confusion at how the government can have a public policy debate about network management when no one except the network operators knows anything about traffic on the net.
NBC filed something with the FCC using the Cache Logic study, done a year after the Pew Internet study, saying that file sharing was dropping and our study showing file sharing was increasing. And the Cache Logic study just came out with a number -- no trends, just that file sharing was 30 to 50 percent of traffic, and NBC uses that number -– way old, no peer review, no methodology -- to say 'You guys, the FCC, have to start policing the network and getting this file sharing off the network.'
All of the data out there is suspect.
The information is vital. Comcast claims that torrents of purloined pop music and movies are filling the internet's tubes -- requiring them to block, divert and dam peer-to-peer traffic. And AT&T says it's going to create technology to detect such sharing by its customers.
Image: Sprint
In Washington, D.C., Congress is once again considering legislating rules for ISPs, while the five-member Federal Trade Commission is publicly wringing its hands over whether to fine or censure Comcast for its BitTorrent blocking and whether to adopt stricter net-neutrality guidelines generally.
For the next couple of weeks, Threat Level is taking a hard look at some of the unsolved mysteries of the internet. This is the first one.
We would love to know if good measurements of P2P traffic are out there or if, indeed, the debate over net neutrality is taking place without the slightest bit of verifiable data.
UPDATE: Ipoque, a P2P traffic management firm, released its own study of internet traffic in 2007, focusing on Germany, Australia, Eastern Europe and Southern Europe.
According to their report, P2P traffic accounted for between 49% and 83 % of internet traffic in these regions. Using deep packet inspection techniques, the company says it could identify the types of files being traded, as well as unique hashes that pinpointed unique files.
For instance in the Middle East, the most popular BitTorrent Audio download was Beyonce's Listen, according to Ipoque. (Does that mean American foreign policy is winning or losing?)
The study is unlikely to please internet scientists, since the data set is not public nor is there much discussion of how the numbers were arrived at.
Photo: Star5112
See Also:
- ISPs' Error Page Ads Let Hackers Hijack Entire Web, Researcher ...
- Commission Ready To Act in Net Neutrality Fight, Says FCC Chief
- FCC Gets an Earful From Open-Net Defenders at Stanford
- Comcast Makes a Deal with BitTorrent
- FCC Chief Promises to Investigate Net Neutrality Complaint Against ...
- Redstate Net Neutrality Flap Jumps To Rush Limbaugh Show
Does Tivo use it's boxes as storage nodes in a P2P network? When you purchase their box, you agree to allow X% of the storage space on the box to be used for promotional material (instead of the TV shows/movies/photos that you record/download). It seems this space could be used for a P2P network. Is it?
This graphs shows PACKETS on the vertical axis. This is in no way representative of traffic percentages.
Packets range from 64bytes to 1500bytes. This graph does not show bandwidth.
most of the argument is null anyway... "valid" web traffic can still include the "pirated" content... south park studios .com now allows streaming videos online... probably more traffic than torrents (being as i doubt it's p2p)...
the problem is that people are using pirated approaches versus businesses offering attractive legal alternatives. once businesses catch up, as south park has, it just becomes part of the "norm"... what we need is a faster pipe! (*cough* ISP's *cough*)
I believe the Ellacoya aproach is quite accurate since it doesn't charecterize traffic based on port numbers. It uses signiatures instead that can identify traffic as P2P regardless if it is encrypted or uses well known ports.
"Comcast claims that torrents of purloined pop music and movies are filling the internet's tubes -- requiring them to block, divert and dam peer-to-peer traffic"
Get out of the business then, Comcast. Put up or shut up. We know you are full of shit if you stay in the business. Nobody is holding a gun to your head to be in the internet traffic business.
LEAVE. GOODBYE.
And don't let the door hit you in the ass.
Right on Publius... whether it be the FCC or some ISP, no one will ever be able to bring down piracy. As soon as someone tries to impose some sort of control or restriction, there's always an exploit and an immediate workaround. Funny how the folks who try to "police" network traffic haven't figured this out yet.
The irony of the whole thing is that the practice of sharing information, in and of itself, has been around way before the internet became popular. Back in the day, no one raised an eyebrow when you would exchange cassette tapes with friends - but now, since technology has made it so much easier, all the sudden it's a concern.
According to the RIAA, all the green cables in the photo are being used to steal music.
maybe bittorrent trackers should opt to release traffic data. it seems their at the forefront of the argument on why the net needs to remain free, open and un-shaped.
a truly independent researcher should start asking them for input. tpb, *nova, etc. will probably be willing to provide accurate data as long as confidentiality is kept.
why waste time with isps? just go to the source.
First, who all really gives a crap?
Comcast whines about the P2P traffic, yet advertises to its customers about streaming music, TV, and movies which take way more traffic than P2P.
The RIAA and others base their P2P traffic on how many people downloaded P2P software, not based on how many people actually use it.
Plus, not all people who use P2P software use it to purloin copyrighted material. There's plenty of "free" stuff out there in P2P land that is legitimate.
Time Warner wants to sell bandwidth based on the user's usage, but also advertises all sorts of streaming media for the user to download with the attendant advertising to pay for the increased bandwidth.
Con-cast has been caught lying about every single aspect of this debate. First they weren't doing anything, then they were just "throttling" p2p traffic. I don't think they are admitting forging tcp reset packets even yet! But we're just supposed to believe them when they tell us it's necessary. BULL! Forget the p2p excuse and force the ISP's to deliver what they promise. If con-cast had invested in their infrastructure instead of simply banking all profits they wouldn't be having a problem now. They caused this problem, make them solve it or make them go away. This "blame game" is old, transparent and childish.
Its just like when the FCC gave the cable company and networks more bandwidth to prepare for HD tv. What did they do with it? They offered more channels with nothing on them. It may sound cliche, but everyone should be writing their senator and congressman. They always vote for what the lobbyists tell them to vote for because they never hear from the public. Just send them an email, they would love to hear what you have to say about this.
Not to defend anyone, but the point of targeting P2P is that it uses the precious upload bandwidth that ISPs don't have/give a lot of. P2P traffic isn't cached in local mirrors, it can't be broadcast, and it isn't multicast (yet?).
How much of the upload link is P2P taking is the question, not how much traffic on the backbone is it using. Comcast doesn't care about the backbone, they care about their own little access tubes.
..ahh how the big corporate lawyers like to play with what they pretend is the free market.
If 90% of broadband is being used for P2P thats fine.. because we're paying for that access. Firms like Comcast advertise their internet at a given speed, with unlimited connectivity, and thats what I pay for. Thats also a service they refuse to provide.
They are marketing and selling a product that in no way lives up to the claims they make about it. I can achieve those magical speeds for some mystical amount of time, minutes, maybe hours. But should I permanently use the bandwidth that I pay for I would be labelled as an abuser of the system and downgraded.
Those companies need to advertise what they are truly offering. Its not unlimited.. it may be a big limit, butnot unlimited. Let them compete with the network they have, not the one the marketers dream up.
Here Here David B!!!!
I won;t let McDonald's sell me 1/2 a hamburger at full price when they advertise a whole hamburger. . . .
I won't prepay $40 for gas and let them give me $20
But we pay $50-$100 a month for broadband and get penalized for expecting to use what we bought.
BTW, Cox cable 'throttles' P2P also
IF I run a P2P clinet like Bittorrent . . .my available bandwidth goes to an infinitely small amount within 20 minutes. I can't check email etc.
It will show I am connected but I can't do anything. Then if I refresh my router and modem. . .still nothing. . . if I turn off P2P and refresh. . .guess what? Full bandwidth.
And I run mulitple comps at home, if ONE runs P2P my entire connection goes to nil. So if I had a person hack my wireless and DL some music on Bittorrent none of my family could check email or play WOW.
Ryan Singel | 
I did a lot of work in this area several years ago. The difficult thing is trying to differentiate traffic based on the port number is inherently flawed. While certain services are traditionally found on specific ports (web services on 80, ssh on 22, etc) there is *no* specific reason why any particular service *has* to run on that port. You can put web services on port 22, ssh on port 2222, ftp on 53621, and so forth. That has a strong tendency to skew data. You also have applications that have control connections on well know ports but spawn connections on ephemeral ports for data transfers - which tend to diffuse the data across a large number of unrelated ports. You also have a lot of bulk file transfers using HTTP which, although its in the spec, sort of blurs the line when it comes to file sharing.
There are some ways to 'fingerprint' the data transfers patterns on a macroscopic level that *may* let you determine rough proportions of different methods of bulk data distribution (p2p, point to point, web, etc) but thats about it.
Are people working with incomplete data? Yes. Is there better information out there? Sort of. Its not super easy to get and still requires a lot of guesswork and interpretation. Such is the nature of the beast (un?) fortunately.