Server Down & No Active Source Connection

[Megaton Cafe Radio]

8 disconnections during fall season. Some 8h long.
More than I can count during January, and February so far has been hell on wheels.

Now again more than 6 server restarts in the last week only (kept all the email messages)
And this time, No active source connection so the radio station is down.

Now guys, as for a service we do pay $170.00 USD per month....
This 100% unacceptable. We are demanding a refund for January and February.
As this is not the 1st time, us clients, have to deal with such a fiasco.

 

[More Support]

We're sorry about the recent issues affecting a your server. We have had a tough time dealing with the recent Spectre / Meltdown vulnerabilities. Unfortunately the patches that were applied by our datacenter introduced a bug which would randomly cause loss of networking on a few of our servers (the worst affected was us2 which megaton cafe happens to be hosted on). The only way to restore networking is to reboot the server which is obviously not ideal and particularly frustrating when you're running a radio server hosting listeners who get disconnected each time. We experimented with some different kernel versions to see if it helped but after a few days the problem would reoccur. These sorts of intermittent problems are the most difficult to diagnose. Late last night we got a reply from the data centre that they have a potential fix but we would have to migrate to different hardware. We emailed everyone on the affected servers and migrated this morning. The migrations took less than 20 minutes and there have been no issues since but only time will tell. We'll continue to monitor the situation of course.

According to our monitoring (https://stats.internet-radio.com/778383566) of the past couple of months us2 has had one downtime event which lasted 8 hours and another for 3 hours. Normally this wouldn't be so long but because of a failure in our reporting mechanism during the night we didn't get the notifications until the following morning. Apologies for this, we have since made our reporting mechanisms more robust. We managed to bring the server back online when an issue occured within 15 minutes during the 3 other unplanned events.

Apart from the recent issues, uptime on us2 (and the other servers) has been very good. During 2017 we had nearly 100% uptime with only 15 minutes of unscheduled downtime for the entire year. Unfortunately though, occasionally a machine will have a problem and we aim to sort it out as quickly as possible so we're sorry we fell short of your (and others) expectations recently.

 

[Megaton Cafe Radio]

The major issues were for 2015-2016. Hence why we have been moved from us1 to us2 which, indeed, was more stable in 2017.
Although, it was a nightmare to transfer everything to new playlists, folders, media and re-organized more than a thousand songs.

As for the recent problems, we need to explain to nearly 70 thousand fans what's going on every time, we lost half of our listeners lately because of that, and the sponsors for our App are not very happy.
We received tons of emails and messages about that, and the fact that it on and off at random. Facebook and Google+ pages to manage, our official website and soon our Instagram as well.

Every single time the server is reset, all the playlists are reset as well, starting from the beginning, repeating the same songs. This is very annoying for our listeners.
And for all this, and let me remind you, $170.00USD/month, still in 2018 we have to deal with apologies and excuses from an anonymous "Staff Member".
We do not even know who we are talking to here, seriously, this is professional? For that price, no it is not. At least have the name of the support team member we are talking to, it would feel more friendly.

In the future, you need a backup plan in case of an emergency, network problems and such, especially for everyone running a real radio station 24/7.

As for now, what can you do for us on this particular matter?.
We do understand you're dealing with problems, so no need for a refund but at least a credit on the next month would be much appreciated considering what we also have to deal with.

 

[More Support]

We remember very well the issues of 2016. Our network provider was being DDOSed and have since invested heavily in improved network infrastructure to mitigate those issues and provide much better networking moving forward hence our near perfect uptime during 2017.

We feel your frustrations, we of course get lots of emails when these issues occur too. It's annoying when you are relying on another company to fix an issue you have little control over. Sorry, we didn't realise that not having signing our forum posts and emails with our names was an issue. It's not been mentioned by anyone else from what I remember. Anyway, I'm Jim (aka "More Support") and Marc is our other support technician (simply "Support").

We are regularly looking into backup plans to completely avoid any downtime but haven't come up with a satisfactory solution yet. Leave it with me to have another look for some more possible ideas based on the recent events.

Did you receive my email yesterday?

 

[Megaton Cafe Radio]

Nice to meet you Jim, much appreciated, thank you sir. I'm Rob (as in my signature).
Names aren't an issue per say, but usually staff members on most chat support or forums on the net will have their names up for their customers. It's just a bit more friendly, professional and less cold id say.

I received your mail today. Thank you again.
I hope you will be able to fix the major issues soon.

For now, the main problem for us is having to deal with such issues for the price we pay.
It is not the price itself, but more like we feel that Radio Stations broadcasting 24-7 don't really have good package with Internet Radio.
I know it is also related to 128kbps songs and that we can use lower, but nowadays 128kbps is an acceptable quality and not something that should be so expensive like 10-15 years ago.

What would really be nice is that you'd have special packages for real Radio Stations like ours with many listeners.
A package with unlimited bandwidth like most internet providers have, or at least something more easy to support for us, as of now it is not the case, and it becomes more and more difficult to sustain each month.

 

[More Support]

Nice to meet you Jim, much appreciated, thank you sir. I'm Rob (as in my signature).
Names aren't an issue per say, but usually staff members on most chat support or forums on the net will have their names up for their customers. It's just a bit more friendly, professional and less cold id say.

I received your mail today. Thank you again.
I hope you will be able to fix the major issues soon.

For now, the main problem for us is having to deal with such issues for the price we pay.
It is not the price itself, but more like we feel that Radio Stations broadcasting 24-7 don't really have good package with Internet Radio.
I know it is also related to 128kbps songs and that we can use lower, but nowadays 128kbps is an acceptable quality and not something that should be so expensive like 10-15 years ago.

What would really be nice is that you'd have special packages for real Radio Stations like ours with many listeners.
A package with unlimited bandwidth like most internet providers have, or at least something more easy to support for us, as of now it is not the case, and it becomes more and more difficult to sustain each month.

Great We haven't had any problems with loss of networking since the migration so fingers crossed I think we're back to normal now. That's a relief!

I'm sorry to hear that our packages don't suit your station's needs so well. We have tried a variety of different packages and billing methods (like pay per listener/bitrate) in the past but our current pay for bandwidth model with unlimited listeners / bitrate is our most popular. Most stations don't like paying for empty listener slots or having listeners blocked because their slots are full.

We don't want to offer unlimited services because in order to do so we would need to compromise on server and network quality and probably support. When we started out over 10 years ago we used cheaper servers and bandwidth but this only resulted in poor network latency, packet loss, worse downtime and often unhappy customers. Radio hosting companies that do offer unlimited hosting at low prices tend to not stick around for very long or won't be interested in hosting popular stations like yours.

You could consider setting up a second server somewhere else and relaying it with the one hosted with us. This would allow you to distribute your listeners over two providers. Unfortunately this wouldn't completely prevent downtime because obviously if your relay source (us) went down so would your second servers source.