All VPhot Users:
George Silvis and I have previously mentioned that images that are uploaded to the VPhot Queue that cannot be plate-solved by Pinpoint cause significant delays in the system. As an example, over the last day, two users uploaded many (100-200+) large images (~12 Mb) that could not be solved. As you can imagine, at 180s each, the queue really hangs up. In the last four months, George has put a lot of programming effort into improving the system BUT if observers continue to upload images that come from new untested systems that Pinpoint cannot solve, we cannot help much. 
So, we again request that new users or regular users who are using new scope/ccd systems either plate-solve their images locally before uploading to VPhot or at least test one image of each image set BEFORE they upload hundreds of images. This is really the best procedure to improve VPhot's operation because it only requires a relatively simple action on your part compared to more difficult programming modifications on our part.
PLEASE try either of these options so we can keep all the users happy! Your help is necessary and much appreciated.
Ken
[quote=MZK]
All VPhot Users:
George Silvis and I have previously mentioned that images that are uploaded to the VPhot Queue that cannot be plate-solved by Pinpoint cause significant delays in the system... As you can imagine, at 180s each, the queue really hangs up. In the last four months, George has put a lot of programming effort into improving the system BUT if observers continue to upload images that come from new untested systems that Pinpoint cannot solve, we cannot help much.
So, we again request that new users or regular users who are using new scope/ccd systems either plate-solve their images locally before uploading to VPhot or at least test one image of each image set BEFORE they upload hundreds of images.
[/quote]
Sorry about always barging in on this discussion, because I am not involved with VPhot development. However, I have a lot of experience in software development over the years, both professionally and on my own.
Yes, it might be "best", from the developer's point of view, to require users to validate all their images first, before uploading them to the queue, but in practice this doesn't work so well. Even though the application is free, and has taken a lot of work by George to develop, it is easy for users to overlook that, and expect it to process all their images automatically without any problems. Imagers are busy people too, and many would just like to submit their data, and let the software do it all for them, and get some sleep :)
On the other hand, given VPhot is mainly a one-man job(?), I can really appreciate the time and effort George has put into making such a wonderful system available to all for free usage.
So, there is an inevitable conflict of interest here between the developer(s) and the users. I think AAVSO has to look into a policy on how to handle the process of software develoment and support. VPhot is obviously a fairly complex system, so do we want to leave it in the hands of a single volunteer? What if that person goes on a lengthy vacation, who will support the system?
I think AAVSO needs to setup an in-house software support/development team. Will and Doc are very good, but they are already very busy with all the system and web stuff, so another core of programming type people need to be in place full-time. To take over support and upgrades to VPhot and other software that may come in the future.
Bottom line is, software that causes frustration and problems for users, is not good public relations for AAVSO!
All that being said, I DO HAVE a quick practical solution to this VPhot queue problem, which I mentioned before: Place a timer in the queue to limit any one job to "x" seconds of plate solving. Once its exceeded, move on to the next image. If the next one from the same user also times-out, then terminate that user's entire run, and send them a message about the problem. The number of seconds for each plate solve "x" can be set by the system operator, based on the average plate-solve times. This will keep the queue going at an acceptable pace for everybody, and single out the users with problem submissions.
So, I hope some of these suggestions are taken to heart. Thank you.
Mike
Good suggestion. But the AAVSO would need to find a way to support the cost of that.
[quote=uis01]
Good suggestion. But the AAVSO would need to find a way to support the cost of that.
[/quote]
Of course. Depends on how much work is involved to maintain, support, upgrade the existing software suite. It's possible a single full-time employee could do it. If they were a programming "generalist" type who can work competently (but necessarily an expert) with a variety of different common languages in use today - C++, Java, .NET, C#, Python, SQL, Perl, etc.
-excellant
Very good points:
-excellant spport today
-risk associated with one technical supporter
-cost to expand support
I believe AAVSO's mission is to provide a consistent approach to data collection utilizing Visual, PEP and CCD photometry:
-Standardized photometry techniques
-Manuals
-Common Star charts
-Standard sequences
Like any organization the AAVSO has to prioritize expenditures in line with its Mission.
VPHOT fits the mission for CCD photometry.
Clear Skies
Mike
A couple comments related to this thread on VPhot support issues.
First, note that VPhot was written by Geir Klingenberg, not by George Silvis. Geir created this phenomenal piece of software under the name Photometrica and later transfered management of it to the AAVSO. The AAVSO sees this as an important benefit for its membership and is working to maintain and improve it. There is a VPhot committee that meets to discuss issues and prioritize enhancement suggestions.
My role on the committee is as the technical maintainer of the system, keeping it up and running. I've stablized and secured the source code, fixed issues that would have stopped it cold (eg, storage constraints and the change in the VSP API) and have worked to improve the operating statistics so the committee can make informed decisions on which bottlenecks to fix and which enhancements to add.
One issue that bothers users is the queue delays. I can see that they occassionally get up to 4 hours. Some of this is sheer volume: VPhot is often handling over 1000 files in a day. This gets aggravated when VPhot is not able to plate solve the images presented to it; it takes longer to test an image and fail than it does to successfully solve an image. As Ken has pointed out, better practice by the users will help this problem. A user who submits 100's of images that all fail plate solution should be a little embarrassed that he is wasting his time and the time of the other users. Anytime you have a new timeseries or changed observing setup, run a handful of images and make sure they solve before pushing in the whole batch.
Could a more sophisticated queue management scheme help? Sure. This is being examined. But It is a bit more complicated to implement than has been suggested.
Meanwhile, we can avoid a Tragedy of the Commons by more careful practice.
If you have specific complaints or feel that VPhot is unuseable in its present form, please send the details to the VPhot committee via VPhotFeedback@gasilvis.net.
George
VPhot maintenace guy
I'm late to this thread but have posted often before, especially with suggestions to improve the whole VPHOT experience. When I could talk with Geir, things got done but the overall administration process seems moribund, especially the one man choke point and the reluctance to put management skills into volunteer management instead of keeping the technical issures inhouse. Nevertheless, here is another attempt to help.
I use Pinpoint myself to plate solve all my images before uploading to VPHOT, saves a lot of time. On the other hand, I have found that if the center of the image as indicated from the FITS header is in the FOV of the image, Pinpoint finds it fairly quickly. On the other hand, if the FITS header value is outside the FOV Pinpoint can take a really long time to plate solve, if it can do it at all. This is a real problem for my acquisition program as my crappy mount can rarely put a new target in the FOV so I must plate solve after a long slew and before I can slew to the final position. My work around is to plate solve with ELBRUS (free, on the INET). If the target is within a 2 deg box centered on the FITS position, ELBRUS will find it quickly. If the target is not inside that box ELBRUS will quickly report no solution.
This could be put to use in VPHOT by either using the plate solution reported by ELBRUS (directly or by feeding it back into Pinpoint) or to reject the image and move on. It seems to me to be be a reasonable burden on the user to require that his/her FITS header be accurate to at least 1 deg to use VPHOT. Were I in charge, I would ban any user who fails that criterion until they showed they could do it consistently. If higher standards are not set we will be plagued by such problems forever.
I don't know if my programming skills are sufficient to implement this fix but I'm reasonably sure such talents exist within the AAVSO community if only the management policies could be amended to take advantage of them (ie, open source).
Jim Roe [ROE]
Since most of my photometric data is taken autonomously, I have to be able to plate solve, using a plate-solving engine like pinpoint, to get an area within my CCD field that contains the variable of interest, comp stars, and a bright enough guide star for my off-axis guider to guide-on. Maybe there should be a requirement to "pre-solve" any image before submitting it to Vphoto....whatever it takes to take a little pain-off using an otherwise marvelous system!
James
p.s. I have to admit, I only use Vphoto to get instrumental magnitudes for Transform-Generator and TA to transform my BVI data!
I spent several years working on the data processing system for the Chandra X-ray observatory. Even though our data came from a single instrument, and was processed by software written by a single organization, building a hardware/software reduction system that really handled all the things that could go wrong was a *tremendous* amount of work, and, ultimately, some manual intervention was still required.
Tom
VPhot Users:
WOW, today was a great night in the queue!
Images uploaded. OK means plate solved; Fault otherwise
User OK Fault Total
Total 1252 1 1253
One failed plate solve out of 1253 images. Thanks for your care.
Ken
PS: George previously created a metric that lets us see how the queue has run and also allows us to see who might be having problems (not shown) so we can help them as needed.
I participate in several networks of telescopes where they automatically send in images as they are taken. Occasionally, sometimes clouds roll in and the roof may or may not close. Image taking continues. Blank images are sometimes sent in, occsionally a few, as in the case of passing clouds and sometimes dozens! Do I need to do something different or should VPhot? Should the telescope proprietor?
Any advice or recommendations would be appreciated.
Lew
Hi Lew:
NO ONE GETS THE BLAME! (Yes, the responsibility but not the blame.)
I will take responsibility (the blame?) for causing your concern.
As I mentioned in my earlier post - "request that new users or regular users who are using new scope/ccd systems either plate-solve their images locally before uploading to VPhot or at least test one image of each image set BEFORE they upload hundreds of images." The "test" is a good practice if you often have problems with your images since you cannot use your images and have wasted your time taking them!
You (and most observers) do not fit into either of these categories. I do not expect that you change your normal upload procedure to prevent some images from slowing the queue occasionally, even if it's a bunch once in a while. Everyone is subject to infrequent cloud interference (or other problems) but not that often because imaging during those conditions is a waste of time and $$. VPhot is efficient enough that it still gets reduction done in a reasonable period of time. The system has not frozen in quite a while. Has anyone noticed? This is especially true as a result of George Silvis' programming efforts over the past four months.
If the VPhot queue always gets slowed to a crawl, we will modify the code to correct that problem as Mike L. has urged. However, that is not the case! That does not mean that I do not get annoyed sometimes with its speed (or lack thereof) and open my mouth (and insert foot!) about it and ask for more care on the part of observers. I use this as a reminder and a teaching tool. I sometimes email the specific user and ask if I can help resolve their difficulty. IF anyone is to blame it is ME.
SO, thank you for being concerned but do not beat yourself up about it. Life is too short.
Ken