Download Word - HSC JIRA

[HSC-1253] reduceFrames.py fails in ProcessFocusTask with NB0921 data Created:
13/May/15 Updated: 03/Jun/15 Resolved: 03/Jun/15
Status:
Project:
Component/s:
Affects
Version/s:
Fix Version/s:
Done
HSC Data Management
hscPipe
None
Type:
Reporter:
Resolution:
Labels:
Remaining
Estimate:
Time Spent:
Original
Estimate:
Story
Hisanori Furusawa
Done
None
Not Specified
Reviewers:
Paul Price
None
Priority:
Assignee:
Votes:
Critical
Hironao Miyatake
0
Not Specified
Not Specified
Description
(1) As of hscPipe 3.7.2, zemax file for NB filters are no set in ProcessFocusTask, leading to the reduceFrames.p
file.
(2) Even when I add a nominal assignment of zemax file to NB filters in hscPipe, I get a RuntimeError killing th
The bottom is the traceback with visit=25800, SSP_Deep_COSMOS, NB0921.
--- a/python/hsc/pipe/tasks/focusTask.py
+++ b/python/hsc/pipe/tasks/focusTask.py
@@ -36,7 +36,11 @@ class ProcessFocusConfig(Config):
"""
super(ProcessFocusConfig, self).setDefaults()
zemaxBase = os.path.join(os.environ["OBS_SUBARU_DIR"], "hsc", "zemax_config%d_0
self.zemax = dict([(f, zemaxBase % n) for f,n in [('g', 9), ('r', 1), ('i', 3),
+
self.zemax = dict([(f, zemaxBase % n) for f,n in [
+
('g', 9), ('r', 1), ('i', 3), ('z', 5), ('y', 7),
+
('N921', 5), ('N816', 3), ('N1010', 7), ('N387', 9), ('N515', 9)
+
]])
+
self.load(os.path.join(os.environ["OBS_SUBARU_DIR"], "config", "hsc", "isr.py")
[0] --- doTansip : end
---
[0] /ana/products.3.7.1/Linux64/numpy/1.8.0/lib/python/site-packages/numpy/core/_methods
empty slice.
[0]
warnings.warn("Mean of empty slice.", RuntimeWarning)
[0] /ana/products.3.7.1/Linux64/numpy/1.8.0/lib/python/site-packages/numpy/core/_methods
value encountered in double_scalars
[0]
ret = ret.dtype.type(ret / rcount)
[0] Warning: converting a masked element to nan.
[0] Warning: converting a masked element to nan.
[0] Warning: converting a masked element to nan.
[0] Warning: converting a masked element to nan.
[0] Warning: converting a masked element to nan.
[0] Warning: converting a masked element to nan.
[0] Warning: converting a masked element to nan.
[0] Warning: converting a masked element to nan.
[0] Warning: converting a masked element to nan.
[0] Warning: converting a masked element to nan.
[0] Warning: converting a masked element to nan.
[0] Warning: converting a masked element to nan.
[0] Warning: converting a masked element to nan.
[0] RuntimeError on hda-mos01:32597 in run: Cannot solve for corrected focus: -- -- [8.2
--> 1000 nan
[0] Traceback (most recent call last):
[0]
File "/ana/products.3.7.1/Linux64/hscPipeBase/3.3.0b_hsc/python/hsc/pipe/base/pool
[0]
return func(*args, **kwargs)
[0]
File
"/data/data01/work/hsc/run/s15a_2014Mar_2015Jul/testAna/deep_cosmos/hscPipe/repo/python/
line 117, in run
[0]
s is not None and s.focus is not None])
[0]
File
"/data/data01/work/hsc/run/s15a_2014Mar_2015Jul/testAna/deep_cosmos/hscPipe/repo/python/
line 247, in solveFocus
[0]
focus = self.focus.measureFocus(focusList, camera, plotName)
[0]
File
"/data/data01/work/hsc/run/s15a_2014Mar_2015Jul/testAna/deep_cosmos/hscPipe/repo/python/
226, in measureFocus
[0]
return getDistanceFromFocus(sources, ccds, dims, zemax, self.config.focus, plotF
[0]
File
"/data/data01/work/hsc/run/s15a_2014Mar_2015Jul/testAna/deep_cosmos/hscPipe/repo/python/
in getDistanceFromFocus
[0]
correctedFocusError, correctedFocusErrorStd = getCorrectedFocusError(reconstruct
reconstructedFocusErrorStd, config.corrCoeff)
[0]
File
"/data/data01/work/hsc/run/s15a_2014Mar_2015Jul/testAna/deep_cosmos/hscPipe/repo/python/
in getCorrectedFocusError
[0]
(f, df, corrCoeff, i, (fff-ff/ff)))
[0] RuntimeError: Cannot solve for corrected focus: -- -- [8.238421, 1.607829, 1.563773,
[0] application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
Comments
Comment by Paul Price [ 13/May/15 ]
Hironao Miyatake, could you please advise?
Comment by Paul Price [ 13/May/15 ]
I expect Hironao Miyatake is busy with the weak lensing meeting this week. Until he's able to
look at this, I suggest disabling the focus measurement for the narrow-band observations.
Comment by Hisanori Furusawa [ 13/May/15 ]
Yes, I agree. I will set doFocus=False. I was wondering if we could have doFocus=False by
default in the pipeline release mainly for scientific data production, as focusTask is mainly for
engineering purposes, and a difficult analysis in nature using the rather vignetted CCDs.
Comment by Paul Price [ 13/May/15 ]
I would like to leave doFocus=True by default, as correlating the focus with the PSF could be
useful in understanding the scientific quality.
Comment by Hisanori Furusawa [ 13/May/15 ]
In that case, we need to make sure any failure in focusTask should not kill the reduceFrames
process to write results incl. CORR files, since It is the problem here that CORR is not written
in this failure of the focusTask.
Comment by Paul Price [ 13/May/15 ]
Good point. I've put a commit on tickets/ of hscPipe to do just that.
price@price-laptop:~/hsc/hscPipe (tickets/HSC-1253=) $ git --no-pager log -patch --reverse origin/master..
commit a8c91a54f0227691ceca20fd12a4432f6f582751
Author: Paul Price <[email protected]>
Date:
Wed May 13 21:27:18 2015 -0400
ProcessExposureTask: catch failures from focus solution
We would like to proceed the best we can if the focus solution fails.
diff --git a/python/hsc/pipe/tasks/processExposure.py
b/python/hsc/pipe/tasks/processExposure.py
index fb04f46..a81f830 100644
--- a/python/hsc/pipe/tasks/processExposure.py
+++ b/python/hsc/pipe/tasks/processExposure.py
@@ -244,7 +244,12 @@ class ProcessExposureTask(BatchPoolTask):
+
+
+
+
+
+
camera = dataRef.get("camera")
plotName = dataRef.get("focusPlot_filename")
focus = self.focus.measureFocus(focusList, camera, plotName)
try:
focus = self.focus.measureFocus(focusList, camera, plotName)
except Exception as e:
self.log.warn("Failed to solve focus: %s" % e)
return None
self.log.info("Focus solution: %s" % (focus,))
metadata = dafBase.PropertyList()
Comment by Hisanori Furusawa [ 14/May/15 ]
Thank you very much for this change. I have confirmed reduceFrames.py goes through the end,
writing resultant files with this change. I also added the above mapping of zemax files for NB
filters in ProcessFocusConfig, and committed to your branch (hscPipe tickets/).
Comment by Hironao Miyatake [ 14/May/15 ]
Sorry that I'm being silent.... I guess the number of stars is not enough to determine the focus,
but I have to look into the image. I'll do that after the lensing meeting.
Comment by Hironao Miyatake [ 27/May/15 ]
Sorry for this late, but I am finally looking into this. It seems that visit=25800 exists in
/tigress/HSC/HSC, but the code cannot find it. Could you look into what is happening?
hscProcessFocus.py /tigress/HSC/HSC --rerun=miyatake/hsc-1253
visit=25800 --clobber-config
--id
2015-05-27T20:20:46: processFocus WARNING: Failed to process {'filter':
'NB0921', 'field': 'SSP_DEEP_COSMOS', 'taiObs': '2015-03-25', 'pointing':
1179, 'ccd': 111, 'visit': 25800, 'dateObs': '2015-03-25', 'expTime':
900.0} (RuntimeError): Unable to retrieve flat for {'taiObs': '2015-03-25',
'pointing': 1179, 'visit': 25800, 'dateObs': '2015-03-25', 'filter':
'NB0921', 'field': 'SSP_DEEP_COSMOS', 'ccd': 111, 'expTime': 900.0}: No
unique lookup for ['calibDate', 'calibVersion'] from {'taiObs': '2015-0325', 'pointing': 1179, 'visit': 25800, 'dateObs': '2015-03-25', 'filter':
'NB0921', 'field': 'SSP_DEEP_COSMOS', 'ccd': 111, 'expTime': 900.0}: 0
matches
Comment by Paul Price [ 27/May/15 ]
The problem was that there was no NB0921 flat. I've created one. Please try again.
Comment by Hironao Miyatake [ 02/Jun/15 ]
Sorry for this late again...
Hmm, hscPipe 3.7.3 can solve this... Do I misunderstand something?
The error Furusawa-san posted means that the focus algorithm failed when applying an
empirical correction to the focus value obtained from rms difference. What happened here is
that there is no focus value to be input to the correction, which indicates that the code could not
calculate the focus value from rms difference.
2015-06-02T21:42:36: processFocus: Focus result for {'taiObs': '2015-0325', 'pointing': 1179, 'visit': 25800, 'dateObs': '2015-03-25', 'filter':
'NB0921', 'field': 'SSP_DEEP_COSMOS', 'expTime': 900.0}:
(0.093694334819477351, 0.027582603877466732, 0.196987369419583,
0.057427754866738207)
Comment by Hisanori Furusawa [ 03/Jun/15 ]
Miyatake-san, Thank you very much for investigating the failure. Recently, I found that my
flatfield for NB0921 has a problem in mask layer, due to my wrong manipulation of the flatfield
files by hands, in which the mask layer was set with BITPIX=-32 rather than 16. This problem
made the processCcd prevent from reading and applying proper mask at the field edge, and this
was the most likely the cause of the failure in focus determination – I am sorry for the
confusion. I have confirmed that with the proper flatfield files, the visit=25800 went through,
finding a close result to your double check.
Comment by Hisanori Furusawa [ 03/Jun/15 ]
Paul-san, would you mind merging hscPipe tickets/ to master if the change is OK to you?
Comment by Hironao Miyatake [ 03/Jun/15 ]
Okay, thank you for the clarification! I am glad to hear it works now.
Comment by Paul Price [ 03/Jun/15 ]
Merged to master. Thanks, both!
Generated at Sat May 06 12:43:26 EDT 2017 using JIRA 6.2.2#6258sha1:65ffb4362589622c100f6488635539584b0f7b98.