Tag Archives: fail

Internet #FAIL at Resurrection Online

The past Sunday morning was not a great at Resurrection Online. First, here is a normal graph of connections from the evening service on June 13:

The blue line is the number of computers that are connected and the green is the estimated worship attendance. The black line at the bottom is the iPhone stream multiplied by 10 so that you can see it on the graph. Here is the graph from Sunday morning:

  1. The initial drop in traffic was the result of a loss of connection between the Resurrection Servers and the Wowza flash servers at Amazon sending out the Flash stream. iPhone stream was not affected, likely as it was on a different subdomain.
  2. Losing the video feed will cause people to reload the page which puts a heavy load on our web server. The deep spike is when the web server was rebooted as people were trying to reconnect.
  3. Uptake around 11:27 was from people seeking to reconnect. The Wowza flash servers were not able to maintain the traffic when people were seeking to reconnect so we did not sustain those who sought to get the connection.
  4. The main player was switched to the iPhone stream around 11:40 and the flash servers restarted.
  5. Restarting the flash servers regained their functionality.

Although the initial failure was out of our control, we are taking steps to mitigate the issue. Sending streams to two different Amazon availability zones with a primary and back up built in as well as continuing to explore other content delivery networks will hopefully move toward preventing this in the future. We have a catastrophic failure like this less than 2% of the time, however we can and need to do better than that.

Thanks to Ian’s quick work on Sunday morning to salvage those that were able to continue worshipping and the entire IT team for the amazing work they do to make Resurrection Online and all of Resurrection’s tech happen well.

UMNS = FAIL on the Constitutional Amendments

United Methodist News Service = FAIL

This morning I was reading news on the USA Today app on my iPhone when I came across this headline, Methodists defeat gay-related membership policy, with the lead sentence:

United Methodists have defeated amendments that would have made church membership open to all Christians regardless of sexual orientation and furthered the creation of a new, U.S.-only governing body, according to the denomination’s news service.

I was surprised to hear the news on the constitutional amendments first through USA Today and not from any denominational source. I was also surprised that the decision had been made so soon, I was under the impression that it would be several more months until the final tally was known as votes were still taking place around the world.

I found the article from the United Methodist News Service under the headline: U.S. conferences disapprove structure proposal. According to the statistics in this article:

  • 29% of annual conferences in the United States have not reported results of voting.
  • 67% of annual conferences in the denomination have not reported results of voting.

According to the article, “For a proposed amendment to be ratified, two-thirds of the aggregate number of voting annual conference members must approve.” The results from the US don’t matter. What matters is the entirety of votes from across the globe. Two-thirds of annual conferences in the denomination haven’t even reported their votes yet.

You have to be joking. The UMNS really published a story with data this incomplete? Unfortunately, yes.

The United Methodist News Service wrote a story too soon from one side of the story with incomplete data.

United Methodist News Service = FAIL

Speedlinking – May 6, 2009