Last summer I was recruited to participate in a data quality task force. Since then, there has been little forward progress aside from a report I compiled. I have now decided to share this document with the membership.
My purpose here is not to start a discussion of specific steps that could be taken to address our data quality - a forum thread is much too unwieldy for that. Rather, the goal is to raise awareness of the situation. The range of anomalies illustrated in the paper represent a challenge to be met with an extended effort, but there are already ideas floating around for short-term actions that would place no burden on the staff. Let me emphasize that *everyone* makes mistakes, me included. I've uploaded some hilariously bad photometry in my time, but I hope that I have corrected it all.
The report (link below) is long but consists primarily of light curves. I think you will find it revealing.
Tom Calderwood
https://cantordust.net/AAVSO/quality_report.pdf
All, take a look.
And it's time to move on this.
Thanks, Tom
George
I was also recruited for this task force, but did not receive any emails.
Thanks for the report, Tom! It should generate lots of discussion.
Arne
This is a wonderful report! Thank you.
I perform two levels of review of my data. First, I check the uncertainty. If it is too high, I try to see why (wrong comps, etc.) and if it cannot come down, I don't upload it.
Next, I check to see if the data matches the pattern of the other observations. If significant difference, and I can't find out why when I review the data, I would seek guidance from other observers since I don't want to upload bad data, but I also don't want to miss a significant event.
I'll get into the study in more depth. However, from a casual read, it appears that some folks do not do that second step. Data is uploaded that does not match the general pattern of observations but the observer did not try to find out why.
Could this second step be automated? By that, I mean that could the system scan new data as it is uploaded, and, if it significantly differs from the general pattern of observations, not upload it till it is examined in some way? Best regards.
Mike
Getting people to scrutinize their data would go a long way towards reducing the amount of anomalous photometry. Again, I don't want to design methods here but there are things that can be done to encourage this. Automatic screening without a human in the loop is a very difficult proposition except for simple parameters like extreme uncertainties or airmasses.
Tom
Tom:
Thank you for investing your time to create this document.
As I read this, one of my more profound realizations is that any time I do let a bad data point slip through, my one bad submission has a much deeper effect than I realized. Each bad submission eats away at the confidence that the astronomical community holds in the AID; my bad data point potentially colors other good data points as being unreliable.
I will ensure that your light curve examples influence the new material being written for the AAVSO CCD/CMOS Photometry Guide that talks to quality checks, self-assessment, and personal quality improvement goals!
- Mark (MMU)
One area in the photometry guides I feel could benefit from much more guidance is the selection of suitable comp stars (color/magnitude matching, single vs. ensemble, proximity in FOV, single-filter vs. transformed, etc). By reading this report, it seems like it warrants a whole chapter, or at the very least a dedicated AAVSO webinar on this topic alone (though much better to have in the guides). At least that's something I'd love to see as more of a novice photometrist.
Keith
Keith:
You have summarized the key criteria for comp selection (i.e., number, location, magnitude, color). That is one of the issues we are discussing/working on in the revised Guide. IMHO, it happens to be the (one of the?) key element(s) to yield accurate magnitudes BUT amazingly it is one of the most contentious issues among digital camera observers, even though to some extent it is not so for visual observers! The proper comp selection and execution has known ability to improve accuracy BUT for many technical/practical reasons (valid or otherwise) MANY observers ignore the "Best Approach".
I personally feel that it is the time to reinforce these criteria / procedures and do what is necessary to make their use 'expected' and easier for photometrists AND subsequently improve our data! I think the Guide should state/recommend/require their use? Time will tell?
Ken
I do like the plotting feature of VPHOT. It is an easy way to spot outliers before submitting the data.
I wonder if an effort could be made to provide VPHOT the ability to transform an ensemble of comps.
Maybe some guidance as to if an untransformed ensemble is more accurate than transform coefficients of some maximum size.
ie; is ensemble on a telescope system with small coefficients more accurate, for whatever reason, than a measurement having one comp and a check having B-V's that don't match the B-V of the target. And since hardly any match, what is the error based on that alone?
Also, could a JPEG of the comp selection accompany all reports? Similar to exoplanet reports?
Ray
"Maybe some guidance as to if an untransformed ensemble is more accurate than transform coefficients of some maximum size.
ie; is ensemble on a telescope system with small coefficients more accurate, for whatever reason, than a measurement having one comp and a check having B-V's that don't match the B-V of the target. And since hardly any match, what is the error based on that alone?"
The answer to the above will depend on the transformation coefficients of a particular camera/filter set/telescope, and can therefore be obtained by experiment (see last paragraph).
Having written that, however, it can be stated that the closer Tv_bv (for example) is to zero (i.e., the closer the transform plot is to a horizontal line), the better non-transformed magnitude determinations will be, since V-v will be 'nearly' independent of B-V. If Tv_bv were in fact zero, V-v would be completely independent of B-V.
In contrast, the higher the value of Tv_bv, the more dependent V-v will be on B-V. Under these circumstances, the closer the B-V values of variable and comp star, the more accurate a non-transformed magnitude will be whereas if there is a large difference between the B-Vs of the variable and comp star a non-transformed magnitude determination will be erroneous.
Since the measured magnitude of a variable star using ensemble photometry can simply be considered as the avarage of the magnitude determinations from each of several comp stars, the accuracy will depend, among other things, on (1) the transformation coefficients of your system, and (2) the range of B-V values of the variable and comp stars.
A simple experiment can provide a direct answer for your system. Image a field of (say) 6 standard stars of various colours. From these, select unique target-comp pairs (from 6 stars you should, I think, be able to get 15 such unique pairs). Determine the non-transformed magnitude of the target star for each pair. Plot the difference between the catalogue magnitude and the measured magnitude of each target (y axis) against each target-comp B-V difference (x axis). A linear function can be applied to the plot having the form y = ax + b. The size of the error of non-transformed magnitude determinations due to target-comp B-V differences can be calculated from the slope 'a'.
Roy
Ray,
VPhot does this with the Two Color Transform tool found in the menu above the images list. To use this your comp sequence file must include a check star and, of course, your VPhot telescope information must include the transforms for the filters you plan to use. It only does two filters at a time.
For B-V transforms check the images list box for a B and a V image taken in the same session, then click on the Two Color Transform tool. The rest is much like the single image photometry tool.
Phil
I hope you'll pardon my comments coming from an outsider. I have examined the opportunity of providing photometry data for several years. I started by using ZAPPER and looking for data errors as my introduction to the research. Doing that was a real learning experience as it showed me how noisy the data was. I really had to wonder what amazing algorithms were used to generate meaningful curves.
As I went through the instructions I realized I did not have the time to fully understand the end-to-end process and really didn't understand what the "transformation" step was. Seemed to be an extra chapter at the back of the instruction PDF. I realized again I was not ready to take this on and continued my "astrophotography for art's sake". Full disclosure, I have a PhD in biochemistry, recently retired from a rewarding career in oncology research but but am also color blind. Art for art's sake is not a good fit, my scientific training and interest has led me to pursue some amateur research projects. AAVSO seemed a good fit with my equipment was certainly of interest as being part of a research group after spending most of my career leading research teams. I had some great researchers and others who just couldn't get things to work in the lab. Based on this excellent report dare I say that there seems to be a lot of people who like making measurements and generating a number, but who really don't fully understand why or what they are doing or contributing to and the ramifications of their submissions. While I understand and greatly appreciate the desire to get people involved and interested in the work of the AAVSO I think in fairness to the organization and the future impact relevance of the AAVSO some competency must be demonstrated and maintained by the observers to allow their contributions to continue to be accepted. It may sound harsh and some folks won't want to adhere, but maybe that will weed out bad submissions as well.
One practical consideration ... when leading an group on human clinical studies we used an electronic data form for physicians to enter recent health and lab data. As soon as the data was submitted a data check was run to see if 1) the result was biologically meaningful (a temperature of >105F was probably an error caught immediately) and 2) if it was meaningful compared to prior data, any change was biologically possible. It seems a simple algorithm to check for basic requirements of submission and comparison to prior data and other observers could either alert the submitter or alert an AAVSO team member of a possible problem.
In conclusion, I'm surprised there has not been more discussion about this problem. It needs to be addressed. Training needs to be updated, codified and periodically reviewed. A system for feedback to observers, with warnings if necessary, must be implemented. All observers should be required to spend at least some time on ZAPPER (I haven't been able to find it by the way) as a reminder and reality check on what they are involved in. It would be also interesting to have a dashboard indication what observations (target and result) have been submitted over the past (day/week/month), perhaps as part of the data submission process as well.
Thank you for your time and dedication. I look forward to figuring out what to do and joining you soon,
NFBascomb
(unedited and reviewed, apologies for errors and omissions)