So there's not much more we can do at this point other than to apologize for the issues. This is not the experience we want our fellow members to have when the book hotels and we're very sorry it happened. We had taken steps before the scheduled open time to increase resources on our web site to handle the increased load and kept monitoring that during the open, but unfortunately this issue was not something directly controlled by us.
I'll try as best I can to answer questions or fill in extra details about what happened. I understand the frustration over not hearing anything and we should have made a post along the lines of "we're not sure what's going on but we're working with Experient on the issue" and directed people to use the phone sooner. We actually had that message crafted to go out when we got word from Experient that the issue had been resolved, so then that took some time to re-draft the new message what was what finally did go out.
Messages like this, especially when we're dealing with one of our partners, do take a little extra time to go out because we need to take extra effort to make sure we're sending the right message. When we said "Otakon and Experient both are sorry for the inconvenience this has caused", we're not putting words in Experient's mouth. They conveyed to us how sorry they were for the issue and we're relaying that message.
Details as to what did happen, very quickly we started getting reports of users getting 404 messages. Several of our staff, including myself, did message some of those reporting issues directly and tried to debug and collected screen shots which we sent to Experient.
it sounds like one or two of the servers out of their web server pool were mis-configured. It took them some time to find and fix that, since the result was only a small portion of users of the site were actually experiencing the issue. So when they were looking at the overall numbers, things were looking good as all their testing was working and large numbers were able to book their rooms. I realize that is little solace if you were in the group of users that did experience the issue.
As someone with a technical background who runs the web servers for Otakon (small scale) and has done so professional (much larger scale), i'll totally admit that I had a head-desk moment when I heard that. We'll certainly be talking to Experient about what the can do in the future to make sure this doesn't happen again.