LibriumArcana.Games

arstechnica.com wrote:The battle over who should pay to carry Netflix traffic is heating up again, and one of the main players blames Verizon's greed for the poor performance that many consumers see when trying to watch streaming video.

Cogent Communications CEO Dave Schaeffer made his case in an interview with Ars yesterday, saying Verizon is refusing to upgrade the infrastructure that carries Internet traffic from one network to another unless outrageous demands for payment are met.

The network connections between Cogent and Verizon, crucial for carrying streaming video and other content to Verizon's home Internet customers, "are full," Schaeffer said. "They are more than full. They are so full that today a significant amount of packets are being dropped between the networks."

First, some background. Cogent is an Internet bandwidth provider that sells transit to Netflix and other companies. When Netflix purchases transit from Cogent, Cogent is responsible for distributing the traffic to all corners of the Internet. But no single company controls the entire Internet. Thus, Cogent must exchange traffic with other network providers, including Verizon.

The connections between Cogent and Verizon take the form of peering. It is a point-to-point connection that doesn't necessarily guarantee passage of traffic to any networks beyond the two involved in the deal. Peering generally happens without any money changing hands, particularly if the two companies involved are of similar size and influence.
Theoretically, the peering arrangement between Cogent and Verizon benefits both. Cogent gets to distribute Netflix traffic, as Netflix pays it to do, and Verizon ensures that its customers, who pay Verizon to access the Internet and are demanding to watch Netflix movies, get the content they request.

Verizon wants to ditch the "settlement-free" peering model and get money from Cogent, arguing that it has to accept far more traffic from Cogent than vice versa because of high-bandwidth applications like Netflix.

Cogent has refused to pay. As negotiations stall, Netflix performance has dropped measurably for months on both Verizon and Comcast. Cogent claims this is because Verizon and others—but especially Verizon—are refusing to upgrade the connections between networks. Cogent points out that Verizon offers its own streaming video services, such as Redbox Instant, and thus has an incentive to harm Netflix traffic.

There are about 11 Cogent/Verizon peering connections in major cities around the country. When peering partners aren't fighting, they typically upgrade the connections (or "ports") when they're about 50 percent full, Cogent says. They can do this by adding ports, adding capacity to ports, or peering in new locations.

With Cogent and Verizon fighting, the upgrades are happening at a glacial pace, according to Schaeffer.

"Once a port hits about 85 percent throughput, you're going to begin to start to drop packets," he said. "Clearly when a port is at 120 or 130 percent [as the Cogent/Verizon ones are] the packet loss is material."

The congestion isn't only happening at peak times, he said. "These ports are so over-congested that they're running in this packet dropping state 22, 24 hours a day. Maybe at four in the morning on Tuesday or something there might be a little bit of headroom," he said.

Slower traffic, and not just on Netflix

Dropped packets can be re-sent, but users could see slower loading times or page load failures. This problem is particularly bad with streaming video because it requires so much bandwidth, with Netflix alone accounting for more than 30 percent of downstream Internet traffic at peak usage times. It does affect all traffic, however.

"If you've got VoIP and you're trying to run Skype, it may not work because the pipe was full, with your neighbor down the street trying to watch Netflix," Schaeffer said. "It's not only the video user. Every Internet user is suffering today in their ability to access all the applications, content, and other users across the Internet."

This isn't the kind of problem consumers can solve by purchasing a faster Internet connection, because they'd be paying only for a faster path to Verizon's network, not a faster path to the rest of the Internet.

In some cases, Verizon has actually purchased and installed the necessary equipment to upgrade ports, but not turned it on, according to Schaeffer. "They actually put it in, so they spent the money, but they just politically have not been willing to turn it on in order to ensure that Netflix will not work as well as Redbox," he said.

When Netflix can't get through to Cogent, it sends traffic through other providers, but Schaeffer claims that "everybody is full."

So what are they fighting over? Schaeffer said Verizon "wants a price that is about 10 times the market price for transit." Again, transit is a service that takes traffic across the entire Internet, whereas peering is just a connection between two networks.

"It's a less robust product at ten times the price," Schaeffer said.

Cogent sells transit for an average price of $1.31 per megabit, though large volume users like Netflix get discounts.

Verizon responds: We don't degrade Netflix to help Redbox

Verizon Senior VP of Public Policy Craig Silliman spoke to Ars today, saying that Cogent is unique in taking such an inflexible stance in negotiations.

There is a wide range of Internet interconnection agreements, he said. "We have settlement-free agreements. There are some ISPs to whom we pay money and there are others who pay money to us," Silliman said. "There is a whole range of commercial options, and they get worked out commercially and smoothly, except for this one ISP who seems to have problems with not just us but a lot of others."

Silliman added that "the whole premise of settlement-free peering is that you have a roughly equal exchange of traffic." He did not say whether any of Netflix's transit providers have agreed to pay Verizon, noting that specific commercial arrangements are confidential.

He also did not reveal how much money Verizon is asking for, saying, "we are open to negotiation for a commercially reasonable solution that works for both parties."

Silliman said he'd have to check with Verizon's engineers to respond to Cogent's allegations regarding ports being more than 100 percent full and Verizon allegedly installing equipment but refusing to turn it on.
He did, however, say, "it is categorically false that we are doing anything to adversely impact Netflix traffic to benefit Redbox."

A recent Wall Street Journal report said that "[e]xecutives at major broadband providers ... privately blame the traffic jam on Netflix's refusal to distribute its traffic more efficiently."

Silliman noted that Cogent has been at the center of public disputes over peering more often than any other company. "It's noteworthy that Cogent is always the one in the middle of these disputes, and they tend to use the media as negotiating leverage," he said.

Cogent digs in for a long standoff

Schaeffer didn't deny that Cogent has fought more public peering battles than anyone. One reason for that is "we only have one business, the Internet. Everybody else, the Internet is a sideline and they've got all these other businesses, but they're kind of internally conflicted. They're not willing to fight."

Level 3 has also been in Cogent's position, fighting Comcast over Netflix traffic in 2010. Level 3 has complained about the sway consumer ISPs hold because they face so little competition and control the only network path to consumers. The company has argued that peering arrangements should be calculated based on "bit miles," the distance traffic is carried and the number of bits carried, regardless of which direction the traffic flows.

However, Level 3 agreed to pay Comcast for peering "under protest." Level 3 also offers a wider variety of business services than Cogent.

Cogent said its battle with Verizon isn't unique. Comcast and Time Warner Cable are demanding payments similar to the ones Verizon wants, and Cogent's connections with those ISPs have hit the "red zone" of 85 percent throughput, Schaeffer alleged.
Level 3 and Comcast declined to comment when contacted by Ars. A Time Warner Cable spokesperson told Ars, "It’s a business discussion with Cogent, and we don’t discuss our business dealings in the press." Executives at major broadband providers, meanwhile, privately blame the traffic jam on Netflix's refusal to distribute its traffic more efficiently.

(UPDATE: Netflix has apparently come to a peering or caching agreement with Comcast.)

Schaeffer said Cogent isn't willing to pay any price for peering with Verizon.

"We believe firmly in bill and keep," he said. "We bill our customers and we connect them to the Internet, and it's our job to deliver to them the best quality possible. Verizon is billing its customers, and it's got a lot more of them and it bills a hell of a lot more, and they're not committed to delivering their customers what they promise the customers they're going to deliver."

Once you pay anything for peering, "it's a slippery slope in two respects," Schaeffer said. "One, everyone wants to get paid then. And then two, once you pay it's like blackmail, they've got you, there's nowhere else to go. They'll just keep raising the price in a market where prices [for transit] are falling."

Netflix attempts to peer directly with ISPs through its Open Connect peering and caching program. British Telecom, Cablevision, Google Fiber, RCN, and others have accepted the offer, but Verizon hasn't. Netflix declined to provide comment to Ars about disputes with Verizon, but Schaeffer said Verizon has demanded money in exchange for peering with Netflix.

"Netflix is free to do whatever they want," Schaeffer said. "I think if Verizon sold them a comparable product at an equal or lower price to Cogent, they should take it. But what Verizon is saying is, 'pay us 10 times as much to get a less robust product, but it's only to get to our customers, and that's the only way you can get to them,' and Netflix understands that's closer to blackmail than a market transaction."

Entrepreneurs trying to become the next Netflix will have a rough time if they and their transit providers have to pay for entry into consumers' homes, Schaeffer said.

"We're committed to driving down the price of Internet," Schaeffer asserted. "If you allow people to impose monopoly taxes on Internet service, it's going to end up resulting in higher prices and lower quality."

Schaeffer said he isn't opposed to senders paying for traffic, but not if it's in addition to consumers paying. "I didn't see Verizon roll out a product that says, 'well if Netflix pays to send content to you, you've got free broadband connectivity,'" he said.

It might get worse, but is there a shred of hope?

The Federal Communications Commission hasn't regulated the peering market, although the topic could come up during proceedings on the proposed Comcast/Time Warner Cable merger. FCC Chairman Tom Wheeler has supported the idea of "two-sided networks" in which ISPs like Verizon can charge both Netflix and their home customers. Wheeler wants to impose some type of net neutrality rules on Internet access, but that would apply only to the "last mile" of connectivity from consumer ISPs to homes, and not the interconnection agreements at issue here.

In Europe last July, the European Commission opened an antitrust probe into whether Orange, Deutsche Telekom, and Telefónica abused market positions in negotiations with content providers. The investigation, which led to a search of those ISPs' offices, was spurred by Cogent's complaints.

Schaeffer thinks the US government will eventually intervene. Individual consumers can't do much to fight, but as peering controversies get more attention and inspire customer outrage, state prosecutors could intervene, he believes.

"I think there will be more press in major mainstream publications. I think there will be more customer ire," he said. "I think service qualities will degrade, and I believe eventually some states' attorney generals will probably step in and sue on behalf of the consumers."