Folks:
Due to a configuration problem, we had a very serious issue with the VPhot server this afternoon. The server crashed, hard, and we are in the midst of rebuilding it. This will take a day or so, and the crash on the Cloud did incur some data loss.
According to VPhot Designer and Programmer Geir Klingenberg, who is spearheading the restoration effort, this is what we can expect:
1) Users will have to "re-register" their telescopes.
2) Saved sequences will need to be redone.
3) The Analysis Logs were lost.
4) We should be able to retrieve about 80-85% of the images and put them back into your directories.
We do have two bits of good news. First, due to weather, we expect the AAVSONet's telescopes are down for a part of this week, so this should be a slow time. Second, since we had to rebuild the server in any event, we are taking the opportunity to rebuild it so that it is FIVE TIMES more powerful!
We believe that this new VPhot will be available again by Thursday afternoon. This wasn't exactly how we were expecting to pull off a VPhot upgrade, but we hope that, in the end, it will be worth it.
We very much apologize for the unexpected downtime, and offer many, many thanks to Geir for expertly spearheading the restoration effort, Ken Menzies, and all the rest of VPhot's users & AAVSO members and contributers who have made it, and AAVSONet, a stunning success over the last couple of years!
If there are any questions, please ask!
-----
Doc Kinne
Thanks for all your efforts guys but...what a disaster.I logged in right now,no images taken 1 hour ago from T5,no sequences,no old images...nothing.At the moment I hope.
What does it mean?In the next days people will start to upload tons of images?wow.
No way at all to make this software avaiable to people in a commercial way?I will pay for it.I insist.It is not possible that every 1-2 month there are problems like this.Server overloaded,too many people using it...
Thanks anyway for you job.I am there with you guys (virtually).Take care
Stefano 'T5' Padovan,PSD
Stefano:
Yes, there was no way around this: this was a bad mis-step, but this is the very first time we've had a problem like this. That being said, the server we are rebuilding is 5x more powerful than the server we had, so we have some realistic hopes that this will alleviate the problems we've had with speed and CPU allocation when both incoming images are being processed as well as when they're being analyzed.
Insofar as making the software commercially available, answering that particular question is above my pay grade, Stefano! :-)
----
Doc Kinne
Hi Stefano:
I just wanted to make a few comments about your observations.
You may be right about the next few days IF VPhot users try to upload "tons of images" in a short period of time. We need to give Geir some time to work his "magic" to restore as many of the images as possible BEFORE anyone does a large upload. Part of the problem with the server may have been caused by a large upload of 6K images all at once!
I would recommend that NO user conduct such a large upload for some time (e.g., weeks?) and then ONLY IF absolutely necessary. If any of my images are not recoverable, I know that I have the original files on my home computer, and I also have all my AAVSO photometry results/reports on my home computer. I try to conduct my photometric reductions on a daily basis and download the reports as well.
It is probably more reasonable to conduct only typical daily uploads of a few hundred images until the upgraded system is created and properly tested. It is hoped that the "more powerful" system will be a siginificant improvement BUT Amazon's technical specifications may NOT tell the whole story. Time will tell. Cross your fingers!
With respect to the possibility of VPhot becoming a commercial product. Geir maintains and enhances VPhot "purely out of love of the challenge" and the "satisfaction" of having created a truely amazing photometry tool for the amateur astronomy community. As he has stated several times, he has NO desire to make this a commercial product, i.e., a business. He has freely given his intellectual property to the AAVSO and continues to enhance and maintain the system for literally no compensation other than because he has "FUN" doing it! I suspect his "fun" may be being sorely tested right now? He is truely an "amazing person"!
I hope, as we all do, that this significant bump in the road is resolved quickly and to everyone's satisfaction. Unfortunately, we each will have some work to re-create our systems (telescopes and sequences). Keep collecting those images! ;-)
Ken
Hi Ken and thanks for the long post.
I have and I think most of us,all the files on my computer and all the datas,so it is not a big deal in case of loss from my side.Once that I have the AAVSO file uploaded for me it is done.Usually when I remember it,I delete old images not needed anymore from the server.I will have to make again the sequences but it is not a big deal,right now I am following about 10 stars so...1 hour or less.Regarding the upload again of lost images of course I will wait weeks and I will upload again just the last days before the crash and just the stars I am following.Just keep us informed
Regarding the commercial side I was talking with Geir in the past,at the Photometrica times,and I know is point of view.Moreover I think there are much more technical issues that I don't know that make not very easy to make the software avaiable for all.It was just an idea to make things easyer at AAVSO but...I am happy like that,the software is really cool.
So guys,thanks for the effort and keep on like this!
Stefano 'T5' Padovan
Thanks for the update, Doc. I know all of you are working hard on the problem. Looking forward to the new VPHOT.
Ed
Ed:
Thank you! This was not exactly the way we were planning on doing an upgrade, but we are rather hoping that it will end up being a worthwhile upgrade.
-----
Doc Kinne
Hi Doc and Geir,
take your time to restore VPhot; it's a very powerful software and it will grow to be even better in the future; ok, we'll have to remake our sequences and re-upload our images, but people should make no problem of that. Accidents happen, that's life, but when I think of the 22 children and 6 adults (from Flanders, Belgium) that died in a bus accident last week in Switzerland ... then the VPhot crash is no big deal anymore. Thanks for all the good work and efforts!
Sic Itur ad Astra
Bart Staels (Flanders, Belgium)
A brief status report before it is time to go to bed:
The server is more or less up and running again, but we have to do some more testing before we declare it ready for use.
I'm also currently re-processing all images that we still have, which is about 35000, hopefully all images from the last few months. So our hope is that you will not need to upload images that have already been uploaded.
Stefano and other iTelescope / SSON users - new images you take are being transferred and stored on the server as usual. They will most likely not be processed before tomorrow, but you will not loose any new images.
More info tomorrow.
Geir
PS: Ken, this IS fun! What would life be without a little action and challenge? :-)
Hi Geir,
yeah,you will have a lot of fun!35000 images to reprocess...
Anyway thanks for your efforts.T5 was not working for me during the crash so no problems and right now is taking an average amount of images so I see no problems and I am not in a hurry.
Thank you and...try to sleep in this days!
Stefano 'T5' PAdovan