connect and disconnect dcc:

lsof -ti:8080 | xargs kill -9

How each data file is saved in scarlet2 project

How each data file is saved in scarlet2 Roman
steps to do scarlet2

To share on group meeting:

Eve: lunch with Phil's student group.
Dan: ~~why not just stack and track stripes of lines, but shift and stack and image subtract? They could be too faint. ~~

03-06

02-24

02-22

-------------------START of Splitting notes based on projects--------------------
Roman Scarlet2:

02-21

From Dan:
Zoomed in residuals for star plot for zero mag correciton plot.
Double check SN procedure is the same with stars. YES

pick one SN that is isolated from the galaxy. Ask Laurens. Use 2017 one that Lauren once told me.

improve the detection. by stacking.

02-20

• Parametrization presentation
  • Having an easier version of model for demo
  • detailed documentaion of how to do it
• Assemble models: got all the files from minitech too.
• plotting SN modeled value vs truth value:

  • I have retrieved truth tables, but they're based on time, not pointitng. I can't find one on one based on time. I'm looking into Lauren's code and see if I can retrieve pointing instead of time.

  • I don't need to to this!!! Just plot all truth as a smooth curve hah. Where to find the code: in qifeng_demo in remake_im_stamp on dcc

  • ^
    ^

  • The points that doesn't look good is very likely because of the zero-point correction is not good: some K band:
    

• Question about Asteroid.

02-19

• Parametrizaion, how to wrap up, ask Ben about design table.
• Model - open model, m00359-k-sma-ft-ld-kf40-4-au.asm.1 assembly file can't open. m00025-k-sma-ft-ld-kf40-cu.prt missing.
• how to load parts into an assembly

#todo
Wait for Arun for stripe line thing, while working with Scarlet2.
Orbit calculator
DCR (differential chromatic Refraction)
how to do the line
Work on what Jake sent us.

Working with Chris and Arun on simuluated images on Roman LSST.

02-18

Issue: Encountered error in installing
export CMAKE_ARGS="-DUSE_OPENMP=OFF"

pip install .

Still unsolved.
Switched to dcc and it works.

But now needs GPU

Checked with Ben, meeting on Thursday

02-17

I want to include the zero-mag correction for light curve from scarlet 2.
My steps: Make sure multiband code work (RomanScarelt2/Simulated_SN_example-Copy4_onRoman_Multiband.ipynb, from local machine).

Current code only takes webbpsf, so it only depends on band. But now I'm using epsf, so I need to do with load psf for each image.

I'm creating a function to read all the epsf I created.
I'm re-creating the image dictionary to include zpmag correction value.
I don't need to have zpmag for the reference image.

I took out the code for previous reading of psf (previously it is based on bands)

band_indices = {band: 0 for band in resampled_psfs_dict.keys()}
ind = band_indices[band]
band_indices[band] += 1
psf_data = resampled_psfs_dict[band]["resampled_psf"]

I also need to build epsf for the reference images (previously I only built for images with SN inside)

Next step from Dan:

• keep working on scarlet2,
• take a look at GitHub - dirac-institute/kbmod: KBMOD (Kernel-Based Moving Object Detection), image subtraction of a moving object.
• Join meeting with Dan and Maryann.

I learned about SN analysis is based on catalogue (structured data), and photometry (images, more like data processing heavy)

Retrieve truth magnitude of the SN with the help of Lauren, and compare.

02-16

I checked the mag-mag difference images from previous dcc jobs. Two of them 20148117_K213_50408_18_zpcorrection_scarlet2 and 20148117_Z087_50133_18_zpcorrection_scarlet2 does not work very well. The others worked.
I checked the total number of the files, and it seems to
I check the log file of 25954846

all jobs are done!! Got all the zero-point correction values for sca_18!! It's in file RomanScarlet2/20148117_200.csv on dcc, and /Users/qifengc/Documents/2_Research.nosync/RomanScarlet2/20148117_200_zpmag.csv on my local computer. Next step is to fix the median value issues, and add this to modeled SN.

Question: how to take into account the uncertainties from this step of finding zeropoint magnitude?

02-14

I try to submit job for build_epsf.sh on dcc. Here is an error I got.

Traceback (most recent call last):

  File "/hpc/group/cosmology/qc59/RomanScarlet2/build_epsf.py", line 292, in <module>

    main(band, pointing, sca)

  File "/hpc/group/cosmology/qc59/RomanScarlet2/build_epsf.py", line 213, in main

    stars = extract_stars(nddata, stars_tbl, size=star_size)  # stars_tbl must be defined

            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/hpc/group/cosmology/qc59/miniconda3/envs/scarlet2/lib/python3.11/site-packages/photutils/psf/epsf_stars.py", line 635, in extract_stars

    stars = [_extract_stars(img, catalogs[0], size=size, use_xy=use_xy)

            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/hpc/group/cosmology/qc59/miniconda3/envs/scarlet2/lib/python3.11/site-packages/photutils/psf/epsf_stars.py", line 635, in <listcomp>

    stars = [_extract_stars(img, catalogs[0], size=size, use_xy=use_xy)

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/hpc/group/cosmology/qc59/miniconda3/envs/scarlet2/lib/python3.11/site-packages/photutils/psf/epsf_stars.py", line 752, in _extract_stars

    large_slc, _ = overlap_slices(data.data.shape, size,

                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/hpc/group/cosmology/qc59/miniconda3/envs/scarlet2/lib/python3.11/site-packages/photutils/utils/cutouts.py", line 17, in _overlap_slices

    slc_lg, slc_sm = overlap_slices(large_array_shape, small_array_shape,

                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/hpc/group/cosmology/qc59/miniconda3/envs/scarlet2/lib/python3.11/site-packages/astropy/nddata/utils.py", line 118, in overlap_slices

    if e_max < 0 or (e_max == 0 and small_array_shape != (0, 0)):

                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Hypothesis: negative values of x, and y are put into extrac_stars(). try add a filter to filter out negative values and see if that works.

What I have done:I add a filter in the filter_stars_for_psf function inside build_epsf.py file.
25957117

New issue:

e "/hpc/group/cosmology/qc59/RomanScarlet2/build_epsf.py", line 1, in <module>

    import matplotlib

ModuleNotFoundError: No module named 'matplotlib'

What is wrong: forgot to activate scarlet2 environment

New job submitted after I activate scarlet2 environment: 25957178

Check log file of 25957178 on 10:28, seems to work.

Check log file of 25957178 , worked.

02-13

JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)

25931149    common zpmag_ta     qc59  R   11:24:06      1 dcc-core-32

25931143    common zpmag_ta     qc59  R   12:55:58      1 dcc-core-33

25931142    common zpmag_ta     qc59  R   18:06:03      1 dcc-core-31

25954978 cosmology build_ep     qc59  R       1:37      1 dcc-cosmology-07

25954846 cosmology zpmag_ta     qc59  R      28:02      1 dcc-cosmology-06

Still waiting:
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)

25931149    common zpmag_ta     qc59  R   10:51:29      1 dcc-core-32

25931143    common zpmag_ta     qc59  R   12:23:21      1 dcc-core-33

25931142    common zpmag_ta     qc59  R   17:33:26      1 dcc-core-31

25916977    common zpmag_ta     qc59  R 1-01:54:57      1 dcc-core-35

JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)

25931150    common zpmag_ta     qc59 PD       0:00      1 (Priority)

25931149    common zpmag_ta     qc59 PD       0:00      1 (Priority)

25931143    common zpmag_ta     qc59 PD       0:00      1 (AssocGrpMemLimit)

25931142    common zpmag_ta     qc59 PD       0:00      1 (AssocGrpMemLimit)

25916978    common zpmag_ta     qc59  R      54:02      1 dcc-core-26

25916977    common zpmag_ta     qc59  R    1:42:29      1 dcc-core-35

25916959    common zpmag_ta     qc59  R    4:40:33      1 dcc-core-46

25897818    common zpmag_ta     qc59  R   14:00:22      1 dcc-core-27
Now submitted until 13, 14 for the job
          25897813    common zpmag_ta     qc59  R    4:38:40      1 dcc-core-28

25897818    common zpmag_ta     qc59  R    4:38:40      1 dcc-core-27

25897822    common zpmag_ta     qc59  R    4:38:40      1 dcc-core-26

25895679    common zpmag_ta     qc59  R    4:57:19      1 dcc-core-35

JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)

25916978    common zpmag_ta     qc59  R      37:04      1 dcc-core-26

25916977    common zpmag_ta     qc59  R    1:25:31      1 dcc-core-35

25916959    common zpmag_ta     qc59  R    4:23:35      1 dcc-core-46

02-12

Check modeling results of DCC from last night

/var/spool/slurmd/job25821703/slurm_script: line 30: write_mag_results.py: command not found

try F184_52283_18 on my local machine: --band F184 --pointing 52283 --sca 18
--band Y106 --pointing 47868 --sca 18

Write .sh file to be able to take a table input:
sbatch zpmag_scarlet2_table.sh --SNID 20148117 --table RomanScarlet2/20148117_200.csv  --start 1 --end 5

Check on these jobs:

             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)

          25871812    common zpmag_ta     qc59 PD       0:00      1 (AssocGrpMemLimit)

          25871806    common zpmag_ta     qc59 PD       0:00      1 (AssocGrpMemLimit)

          25871771    common zpmag_ta     qc59  R       5:35      1 dcc-core-34

          25845976 cosmology zpmag-sc     qc59  R    8:14:24      1 dcc-cosmology-03

          25845652 cosmology zpmag-sc     qc59  R    8:21:50      1 dcc-cosmology-02

          25845257 cosmology zpmag-sc     qc59  R    8:36:10      1 dcc-cosmology-01

Parametrization

Parametrized:

Leo stat:

Measure from the bottom plane of the plate. detector side.
Where it eats up the measurement

Lens 2:

parametrize from this plane, sky-side

The part that eat up the distance

lens 3:

Where I model the distance, I measured from sky-side

Where it eats up the distance:

02-11

now the code is fully executable, just by giving the pointing and the band.

But the second results doesn't look good....

Retrying the first one.

Note: some sources are detected wrongly

After retrying the first one:

The first one also looks weird

Found the issue:
Since I'm putting flux_sc2 based on idx, however, my original file does not exclude galaxy type, so there is a mismatch between the index of the result and its index in the table.

Now Fixed!!

Now try, writing it to a new csv file that only has the stars, when constructing the observations.
also, how to make the process faster.

Wrote it in .sh file
--SNID 20148117 --band R062 --pointing 50463 --sca 18
--SNID 20148117 --band Y106 --pointing 52188 --sca 18
--SNID 20148117 --band K213 --pointing 52338 --sca 18

job submitted to dcc

(scarlet2) qc59@dcc-login-01  **/hpc/group/cosmology/qc59 $** squeue -u qc59

             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)

          25821703 cosmology zpmag-sc     qc59 PD       0:00      1 (Priority)

          25820580 cosmology zpmag-sc     qc59  R      50:22      1 dcc-cosmology-08

          25821496 gpu-commo zpmag-sc     qc59  R       5:06      1 dcc-core-gpu-08

          25820963 gpu-commo zpmag-sc     qc59  R      38:45      1 dcc-core-gpu-41

to report to Ben

Total length: 898.02mm
the lens 1 will shift by 0.6632 mm

Equation should be setting a dimension to certain values

02-10

Modeling results:

29 of the failed, see detailed logs: Modeling stars with Scarlet2 log
#toFix Take care of the failed cases

My plot for difference of zero-mag correction:

Take magnitude from 19-21 dots for magnitude correction.

Yellow dashed line is the magnitude correction: zero_point_correction: 6.05581382068202

Write results to files to read and plot

Use write_mag_results.py to write the results

this strange matplotlib issue?

/opt/homebrew/anaconda3/envs/scarlet2/bin/python /Users/qifengc/Documents/2_Research.nosync/RomanScarlet2/plot_mag_mag.py
Traceback (most recent call last):
  File "/Users/qifengc/Documents/2_Research.nosync/RomanScarlet2/plot_mag_mag.py", line 8, in <module>
    plt.figure(figsize=(8, 8))
  File "/opt/homebrew/anaconda3/envs/scarlet2/lib/python3.10/site-packages/matplotlib/pyplot.py", line 1027, in figure
    manager = new_figure_manager(
  File "/opt/homebrew/anaconda3/envs/scarlet2/lib/python3.10/site-packages/matplotlib/pyplot.py", line 549, in new_figure_manager
    _warn_if_gui_out_of_main_thread()
  File "/opt/homebrew/anaconda3/envs/scarlet2/lib/python3.10/site-packages/matplotlib/pyplot.py", line 526, in _warn_if_gui_out_of_main_thread
    canvas_class = cast(type[FigureCanvasBase], _get_backend_mod().FigureCanvas)
  File "/opt/homebrew/anaconda3/envs/scarlet2/lib/python3.10/site-packages/matplotlib/pyplot.py", line 358, in _get_backend_mod
    switch_backend(rcParams._get("backend"))
  File "/opt/homebrew/anaconda3/envs/scarlet2/lib/python3.10/site-packages/matplotlib/pyplot.py", line 416, in switch_backend
    canvas_class = module.FigureCanvas
AttributeError: module 'backend_interagg' has no attribute 'FigureCanvas'. Did you mean: 'FigureCanvasAgg'?

Process finished with exit code 1

• Solutions:
  • This error means that matplotlib is trying to use a backend called "interagg"—which does not have the attribute FigureCanvas—instead of a proper backend like "Agg" or "TkAgg". In your case, you likely want a non-interactive backend (e.g. "Agg") to generate and save plots.

  import matplotlib
  matplotlib.use("Agg")  # or "TkAgg" if you prefer an interactive backend
  import matplotlib.pyplot as plt
  
  # Now the rest of your plotting code
  plt.figure(figsize=(8, 8))
  # ... your plotting commands ...
  plt.savefig("output_plot.png")
  

the full process of taking zero-magnitude correction into SN modeling:

Now it's clear!!!

• prepare psf for each image (process all the stars)
  • extract stars to perform psf
  • write it in an executable python file
• prepare zero_mag_shift for each object (modeling all the stars)
  • extract all the stars: need to filter out some stars first.
  • prepare truth mag for all the stars : already there, but it's only for one image 20148117_all_sources_with_truth.csv I should get one file per image from DCC.
  • prepare scarlet2 objects for all the stars: I need observations, observations_sc2, and channels_sc2
    • Note!! Now it's global background subtraciton, before making cutout.
  • run the modeling (make main_loop and model_one py file able to take other inputs. )
  • Calculate zero_mag_shift (how to figure out the linear part automatically?), store it in a cvs file, associate it to each image.
• Model SN
  • scarlet 2 observation objects for SN
  • Modeling SN, read the information of zero_mag_shift, add the shift to the final output. Store both the magnitude without zero shift and with zero shift, and plot them.

1. Write a code to get information of all sources from DCC: worked:
   Run it on DCC:
   For single usage:
   ./get_sources_info.py --band Y106 --pointing 50188 --sca 18
   For multiple: write it into an input file.
    ./get_sources_info.py --input-file 20148117_200.cs
   It retrieves all information of sources per image, including calculating truth_mag for each object, and saved to a csv file.

02-09

Write Roman Scarlet code in separeted loop:

model_one.py: do model for one observation
main_loop.py: use subprocess to loop.

result = subprocess.run(  
    ['python', 'model_one.py', str(idx)],  
    capture_output=True,  # Capture output if you want to print it  
    text=True  
)

It takes a lot of time to run.... And there are some errors for some of the modeling.

Some errors:

Starting modeling for index 3 ...
Modeling for index 3 failed:
WARNING: FITSFixedWarning: The WCS transformation has more axes (2) than the image it is associated with (0) [astropy.wcs.wcs]
WARNING: FITSFixedWarning: 'datfix' made the change 'Changed DATE-OBS from '2030-05-29 12:57:36' to '2030-05-29T12:57:36''. [astropy.wcs.wcs]
/Users/qifengc/Documents/2_Research.nosync/RomanScarlet2/model_one.py:268: UserWarning: A JAX array is being set as static! This can result in unexpected behavior and is usually a mistake to do.
  obs_sc2 = scarlet2.Observation(jnp.asarray(obs.data).copy(), jnp.asarray(obs.weights).copy(),
Traceback (most recent call last):
  File "/Users/qifengc/Documents/2_Research.nosync/RomanScarlet2/model_one.py", line 493, in <module>
    main()
  File "/Users/qifengc/Documents/2_Research.nosync/RomanScarlet2/model_one.py", line 280, in main
    i_star = find_closest_to_image_center_index(ra_dec, wcs_ref, cutout_size[0])
  File "/Users/qifengc/Documents/2_Research.nosync/RomanScarlet2/model_one.py", line 167, in find_closest_to_image_center_index
    coords = SkyCoord(ra=coord_array[:, 0] * u.degree, dec=coord_array[:, 1] * u.degree)
IndexError: index 0 is out of bounds for axis 1 with size 0

Starting modeling for index 4 ...
Modeling for index 4 failed:
WARNING: FITSFixedWarning: The WCS transformation has more axes (2) than the image it is associated with (0) [astropy.wcs.wcs]
WARNING: FITSFixedWarning: 'datfix' made the change 'Changed DATE-OBS from '2030-05-29 12:57:36' to '2030-05-29T12:57:36''. [astropy.wcs.wcs]
/Users/qifengc/Documents/2_Research.nosync/RomanScarlet2/model_one.py:268: UserWarning: A JAX array is being set as static! This can result in unexpected behavior and is usually a mistake to do.
  obs_sc2 = scarlet2.Observation(jnp.asarray(obs.data).copy(), jnp.asarray(obs.weights).copy(),
Traceback (most recent call last):
  File "/Users/qifengc/Documents/2_Research.nosync/RomanScarlet2/model_one.py", line 493, in <module>
    main()
  File "/Users/qifengc/Documents/2_Research.nosync/RomanScarlet2/model_one.py", line 327, in main
    specturm, morph_init = initialization.from_gaussian_moments(observations_sc2, centerpix)
  File "/Users/qifengc/Documents/2_Research.nosync/RomanScarlet2/scarlet2/initialization.py", line 222, in from_gaussian_moments
    morph = GaussianMorphology.from_moments(g)
  File "/Users/qifengc/Documents/2_Research.nosync/RomanScarlet2/scarlet2/morphology.py", line 138, in from_moments
    raise ValueError(
ValueError: Gaussian morphology not possible with size=nan, and ellipticity=[ 0.13045451 -1.3021326 ]!

Starting modeling for index 9 ...
Modeling for index 9 failed:
WARNING: FITSFixedWarning: The WCS transformation has more axes (2) than the image it is associated with (0) [astropy.wcs.wcs]
WARNING: FITSFixedWarning: 'datfix' made the change 'Changed DATE-OBS from '2030-05-29 12:57:36' to '2030-05-29T12:57:36''. [astropy.wcs.wcs]
/Users/qifengc/Documents/2_Research.nosync/RomanScarlet2/model_one.py:268: UserWarning: A JAX array is being set as static! This can result in unexpected behavior and is usually a mistake to do.
  obs_sc2 = scarlet2.Observation(jnp.asarray(obs.data).copy(), jnp.asarray(obs.weights).copy(),
Traceback (most recent call last):
  File "/Users/qifengc/Documents/2_Research.nosync/RomanScarlet2/model_one.py", line 493, in <module>
    main()
  File "/Users/qifengc/Documents/2_Research.nosync/RomanScarlet2/model_one.py", line 280, in main
    i_star = find_closest_to_image_center_index(ra_dec, wcs_ref, cutout_size[0])
  File "/Users/qifengc/Documents/2_Research.nosync/RomanScarlet2/model_one.py", line 167, in find_closest_to_image_center_index
    coords = SkyCoord(ra=coord_array[:, 0] * u.degree, dec=coord_array[:, 1] * u.degree)
IndexError: index 0 is out of bounds for axis 1 with size 0

02-07

Starts to warm up!!!

check with Ben (probably next week)
Ask questions about the cool down page
Get the coil thing work
Start reviewing the wiring inside fridge.

Go in the shop to check.
Install the coil.

Thanks.
Copy most of them. Leave out grafana, theomormeter stuff, file path will be different.
Look at the scripts, and see if the procedures are there.

02-06

Pulse tube compressor.
Ask question: what are the parts that when we're operating the system, we need to do it manually, and I would like to go through the process.

Very interesting ideas about the

02-05

Helion Gas leak vs water leak.

01-31

After fitting Wow it seems to be better

Purchase of filter, heat exchanger arrive on time, hose barb, they'll put things back as planned.
Some issue and they

Helmholtz coil
Wood part, machine shop next door, metal part: from mini tech.
Show our plan with Eve

Make template for cooldown log

01-30

The loop worked after I enlarge the size of psf. But it also won't be able to

Meeting with Ben:
Ben's "flange", from left side of the yellow thing.

Measure the total length from the left side of the yellow thing to the hexigon.

01-29

To do:

• First try the epsf with the modeling.
• Debug the stars that doesn't go through.

Save epsf: using pickle file to incude all informations.

Error for star index 0:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[111], line 62
     19 #         if i==indtransient: 
     20 
     21 #             flux = np.copy(np.asarray(initialization.pixel_spectrum(observations_sc2, centerpix).data))
   (...)
     58                    
     59 #         else:
     61     flux = 1.8*jnp.asarray(np.asarray(initialization.pixel_spectrum(observations_sc2, centerpix).data))
---> 62     specturm, morph_init = initialization.from_gaussian_moments(observations_sc2, centerpix)
     63     print(type(morph_init))
     64     morph_init = np.asarray(morph_init.data, dtype=np.float64)

File ~/Documents/2_Research.nosync/RomanScarlet2/scarlet2/initialization.py:222, in from_gaussian_moments(obs, center, min_size, delta_size, min_snr, min_corr, min_value, return_array)
    219     g[key] = jnp.median(g[key])  # this is not SNR weighted nor consistent aross different moments, but works(?)
    221 # create morphology and evaluate at center
--> 222 morph = GaussianMorphology.from_moments(g)
    223 morph = morph()
    224 spectrum /= morph.sum()

File ~/Documents/2_Research.nosync/RomanScarlet2/scarlet2/morphology.py:138, in GaussianMorphology.from_moments(g)
    136     morph = GaussianMorphology(T, ellipticity)
    137 else:
--> 138     raise ValueError(
    139         f"Gaussian morphology not possible with size={T}, and ellipticity={ellipticity}!")
    140 return morph

ValueError: Gaussian morphology not possible with size=nan, and ellipticity=[ 3.6470506 -0.6407697]!

Debugging of this

Pixel coord issue?
Debugging with the .py file:
scarlet2/scarlet2/initialization.py at 604d5d2a61d065fe0e96b3445006a8b4e01c906b · pmelchior/scarlet2 · GitHub
Looks very interesting...

Increasing the size of psf (when constructing epsf) works now. But I don't know why.

01-28

Check with Jenna on buying filter
From Lauren:

• Write the code in a for loop (for doing models)
• plot difference of my mag - truth_mag on the y axis, and truth_mag on the left axis, to see if they center on 0
• Apply the new psf, and test on the model, and do the same plot as the previous step, to see if that changes something.

#notes Exclude_percentile has been set from 20 to 30 when processing all 168 stars.
#todo Maybe also just use those stars that are far from each other to calculate the magnitude?

I mostly work on the loop thing. It doesn't work through. Some stars will raise errors. And there will be different errors.
For the buying part of filter, we probably need 4 adapters...

01-27

To do:

• turn the modeling process of a star into an iterrable code
• fix the psf thing with Lauren
  Related code:
  Local: Simulated_SN_example-Copy4_onRoman_correction_star
  Dcc: Together-correction-for-star

SN:

Ask Lauren about PSF

• Use phtoutils epsf: Building an effective Point Spread Function (ePSF) — photutils 2.1.1.dev7+g88a65cb
• Investigate on webbpsf's code and see if they uses galsim

My current thoughts: build it myself. But it will take some time. Check in with Dan later.

Build it from the star list I have before, select out those stars whose peak value is big enough, and are separated from each other.
Some of the stars:

My built psf:

lab:

Need an alternative cooling water scheme, by using a filter.

2. Heat Exchanger Delay & Alternative Cooling Setup
   • The heat exchanger, which is crucial for regulating cooling water, is delayed.
   • To keep the system running, we are exploring directly using the building’s cooling water instead.
   • Concern: Building-supplied water may not be clean enough, leading to the suggestion of installing a whole house filter before the water enters the system.
3. Whole House Filter Considerations
   • A whole house filter can remove sediments, debris, and contaminants from the inlet cooling water to protect our system.
   • Oversizing the filter is recommended to minimize pressure drop and avoid restricting water flow.
   • Since we don’t have pressure readings from the design phase, we aim for a filter with <3 psi pressure drop at 5 GPM, but the lower, the better.
   • Balancing valves will need to be adjusted post-installation to ensure proper water distribution, particularly to the compressor and GHS.
4. Process Cooling Water (PCW) Lines & Their Role
   • The PCW lines are fully installed and are part of the building’s cooling water infrastructure.
   • Testing of the PCW lines was proposed, but it is not possible yet due to missing hoses for the final connections.
   • The hoses will need MNPT to FNPT adapters for proper connections.
   • A pressure test is scheduled to verify if the PCW lines can handle water pressure.
5. Relation Between PCW Lines & Whole House Filter
   • PCW lines deliver cooling water, but they do not ensure water quality.
   • If PCW water is used directly, the whole house filter may still be necessary.
   • If the heat exchanger were available, it might reduce or eliminate the need for a filter.
   

• Need a adapter for one side?
• How to check the pressure drop?
  Current list I found: McMaster-Carr

01-24

Read star catalogue with filters of pixels: working file: Together-correction-for-star.ipynb on dcc/remake_image_stamp

• filter out x, y larger than 4088 -- modified: # Chose 4088 - 80, and 0+80 for filter, so that I can at least create a 80by80 cutout
• Work on modeling for one star, and one image and it's finished!
• Comparing the first three star magnitude between Truth mag and mag derived by Scarlet.

Modeling of the star:

My thoughts:
I’m thinking about modeling the star with one image (since we need to derive the magnitude correction per image), with your pipeline, but set the SN start time to be infinity (meaning the varying point source will never get turned on).

Constructed observation objects.
I don't need source detection I think... But I need the weight making process. so don't skip the cell

Why the detected source has a different ra and dec from the original ra and dec?

Finished model with stars:
Just took off all the transient part. And model with a static source.

Modeled with 3 stars, the trend is good, but not too precise. Need to fix psf, asked Lauren, waiting for response.
from scarlet model:
array([14.1741599])
array([14.81621139])
array([18.17318572])

from the truth catalogue model:
|20.39686693619874|
|20.870980128376488|
|23.91935988339908|

Next step:

• correct psf
• write it in a loop

Simulated_SN_example-Copy4_onRoman_correction_star

01-23

Working on retrieving Roman images:
snid = 20172782
result = req.post( f'{server_url}/findromanimages/id=({snid})' )
Not working....
✅should be findtransients, instead of find images.
Switch to cloud:

• Issue with running jupyter: DCC Jupyter Issues

Understanding the zero magnitude correction process Cosmology notebook#Magnitude Correction

Started to read star catalogue, worked. Retried magnitude zero point.
#notes My first image to conduct magnitude correction
table_all = read_truth_txt(path=None, band="Y106", pointing=50188, sca=18)

Some issue with background subtractors:
❌: switch to what Laurence's default background estimator:
took background subtraction out of photometry code. now requires pre-… · Roman-Supernova-PIT/phrosty@e13f32d · GitHub
Does not work..
✅Tried increase exclude_percentile = 20, default is 0. And it works! This removes the lowest-intensity pixels (in each box_size region) before computing the background.

First 10 stars:

01-22

Refresh myself about the project.
Measuerment from myside:

Lower 4K tube:
Overall length: 583.28 mm or 22.964'' (using measurement tool)
RD21@Annotations
Flange at front end: 0.492'' (from part’s dimension)Upper 4K tube:
"flange4": D6@Sketch1@Lower optics tube top flange
Overall length: 294.24mm or 11.584'' (using measurement tool)
RD20 @Annotations


flange1: lens 1 cell, bottm (the holder? flange)
Note from ben:
It's called lens 1 bottom
Flange always refers to the welded ones

01-21

Meet with Dan and Eve. Erik's defense. Figured out a plan

01-20

Even worse, after correcting for zero point:

Constructing meeting agenda with Eve and Dan Meeting with Eve and Dan

01-18

To do:

• add zero point magnitude
• add mjd on and mjd off from the package.
• convert the code in a way that it can be submitted to dcc job.

Zero point issue

• from header: [ZPTMAG]
• q = mcmc.get_samples()['spectrum:'+str(indtransient)]
  shape: column: each band (epoch information + band), row: one sampling

add start_mjd and end_mjd, from the pacakge

• roman-desc-simdex/documentation.ipynb at main · Roman-Supernova-PIT/roman-desc-simdex · GitHub
• Interestingly, now I have two images before SN starts (previously I though I only have one images associated with galaxy)

Result: looks even worse?

Next: check with truth file, to see what's the true value of it?

Explore my research area:

• What makes CCAT better than ALMA?

• How is Radio images different from opticals/infared ones

• How would SN detection benefit from radio images?

• How does the data processing of radio images differ from optical images?

• Cross-check the extracted lightcurves from Roman/LSST with the Radio images.

• Study how SN will be benefited from Radio images, and if beneficial, we can have some SN observation strategies in CCAT, and study what kind of strategies that can be.

• why current SN images are mostly from infrared or optical cameras?

  • reach peak luminosity in optical band
  • rich spectrum information in optical bands, to distinguish different classes

• How is different classes of SN defined?

• What are different distance ladder?

• How does CMB help with determining H0?

• How do we know that the current SN are not lensed by their host galaxy? Or other foreground galaxy?
  #notes
  Absolute magnitude and apparent magnitude:

1. Convert Flux to Apparent Magnitude

The apparent magnitude mm can be calculated using the zero-point magnitude ZPTMAGZPTMAG and the measured flux FF using the standard magnitude equation:

m=ZPTMAG−2.5log⁡10(F)

2. Convert Apparent Magnitude to Absolute Magnitude

Once you have the apparent magnitude mm, you can calculate the absolute magnitude MM using the distance modulusformula:

M=m−5log⁡10(D)+5

where:

• DD is the luminosity distance to the object in parsecs (pc).
• MM is the absolute magnitude, which represents the object's intrinsic brightness as if it were at a standard distance of 10 parsecs.

01-12 - 01-16

Roman workshop, note: Roman Workshop AAS
AAS 245 Conference, note:AAS 245 Winter in DC

01-09

Preparing for posters

01-08

Try this object to compare with Lauren: roman-desc-simdex.lbl.gov/findtransients/id=20172782
id = 20172782

A useful link from Lauren:
roman-desc-simdex/documentation.ipynb at main · Roman-Supernova-PIT/roman-desc-simdex · GitHub

{"av":[-9.0],"dec":[-44.80718106491529],"end_mjd":[62881.0],"gentype":[10],"healpix":[10430],"host_dec":[-44.8073],"host_id":[10430000050524],"host_mag_f":[22.179],"host_mag_g":[23.763],"host_mag_i":[22.871],"host_ra":[7.550449],"host_sn_sep":[1.7006677],"id":[20172782],"lens_dmu":[0.0],"lens_dmu_applied":[false],"model_name":["SALT3.NIR_WAVEEXT"],"model_params":[{"salt2_alpha":0.15000000596046448,"salt2_beta":3.0999999046325684,"salt2_c":0.1403409242630005,"salt2_gammaDM":0.0,"salt2_mB":22.55720329284668,"salt2_x0":1.702625013422221e-05,"salt2_x1":-0.3297336995601654,"template_index":0.0}],"mw_ebv":[0.0],"mw_extinction_applied":[false],"peak_mag_f":[23.901009],"peak_mag_g":[23.162024],"peak_mag_i":[22.261768],"peak_mjd":[62476.508],"ra":[7.551093401915147],"rv":[-9.0],"start_mjd":[62450.0],"v_pec":[0.0],"z_cmb":[0.3601]}

id=2014:

Steps:

• generate a table that is associate with the SN (on dcc) and save as cvs
• generate a table that is associate with the galaxy (on dcc) and save as cvs
• download the fits file (download fits.sh)

Wrote .py and .sh file so that the information retrieving process is done by dcc sbatch job.

Learned some useful command for vim

go to:
start of the file: gg
end of the file: G
some lines: 45G

start of a line: 0
end of a line: $

Select:
V: line
v: word, shift, jump to next word
u: undo last modification

Indent:>>

Move page:
• Ctrl-d (down): Moves the cursor half a screen down.

• Ctrl-f (forward): Moves the cursor one full screen forward (to the next page).

• Ctrl-u (up): Moves the cursor half a screen up.

• Ctrl-b (backward): Moves the cursor one full screen backward (to the previous page).

From Lauren's code to retrieve data files: if checked oid in truth, no object will be returned. Temporarly shut down this part.

01-06

tried multiple band

01-04

Treat the star as a transient:
The flux looks constant, but it's larger than just modeling it as a galaxy

What else I can try:

Figure out the meaning of scale:

fac = np.sum(scene_.sources[indtransient].morphology(), axis=(-2, -1))
fluxes.append(q1[2][i] * fac)
fluxerrors.append(np.abs(q1[4][i] - q1[0][i]) * fac)

• scene_.sources[indtransient].morphology() returns a 2D array where each element represents the fraction of the source’s flux in a particular pixel of the image.

• This morphology is normalized such that the integral (or sum) over all pixels corresponds to the total flux of the source.

! the two fac (from Charlotte's data and from my data is the same: 3.6288116, something must be wrong)
Even my morphology matrix is the same! Should they be the same?
Some potential reasoning:

• If the initial morphology guesses for the source are the same and the optimization process does not have sufficient data to refine the morphologies, the output morphologies might remain identical.
• If the source is heavily blended with another source (e.g., a host galaxy), the optimization might converge to the same morphology dominated by the blend.

In the HSC_AGN_example.ipynb book, they skipped some poor images: Can further investigate

        if np.sum(weight_hsc==0)>0.001*weight_hsc.shape[-2]*weight_hsc.shape[-1]:
            print('Poor image, skipping')
            continue

wrote a script to download files in a row

./download_fits_file.sh under RomanScarlet2 folder.

use ssh passkey to avoid typing password and MFA all the time

Login - Duke Compute Cluster
I have my previous passkey from illinois. I used that and succeeded.

Now it can automatically download files. Yeah!!!

shut down a few bands from the LSST's data, still looks fine....

trying to add more band images. but be minded that I haven't added any images for the galaxy. [Done]

#todo generate a table of information that's for galaxies.

try background subtraction before constructing the scarlet object.

I have subtracted data ready.

#notes Images from different bands are not stacked into one.

Multiple bands results: