Debugging

Debugging MESA requires effort and tenacity. There is no single approach that is applicable in all circumstances. This page collects information on frequently useful tools and techniques.

General Techniques

Using a debugger

The MESA SDK contains the GNU debugger gdb. Detailed instructions for using gdb are beyond the scope of the documentation, but it allows you to interactively set breakpoints, step through code, and inspect values.

Using pgstar

The built in pgstar plotting is itself an invaluable a debugging tool. Something may jump out from a plot as strange behavior before it shows up as a problem in the run. You may notice that an issue begins when the model reaches certain conditions and that can provide a useful guide to the problem. After you identify an issue and configure some useful plots, consider setting pgstar_interval = 1 and pause = .true. so that you can step through the model timestep-by-timestep.

Using a profiler

Troubleshooting performance bugs or memory leaks benefits from specialized tools (see Profiling).

Diagnosing Solver Struggles

Identifying when the solver is having difficulties and understanding the root cause is an important subset of MESA debugging. When you see a MESA model experiencing problems (e.g., taking unexpectedly small timesteps due to many retries), you should at the least report the issue to the developers’ list and ideally investigate it a bit yourself. Paying attention to such problems has proved a valuable way of identifying errors and inconsistencies within MESA.

Diagnosing solver struggles primarily relies on internal MESA tooling. The most common cause of solver struggles is incorrect/inaccurate derivatives. This document will illustrate how to go from a failing model down into the depths of the code in order to identify and fix the problem.

Step -1: Introduce a bug

For the purpose of this tutorial, we will introduce a bug in MESA. In more typical circumstances, you or someone else will have kindly already taken care of this step.

Navigate to the subroutine Get_kap_Results in kap/private/kap_eval.f90. After the call to Get_kap_Results_blend_T, add

dlnkap_rad_dlnRho = -dlnkap_rad_dlnRho

thereby making the derivatives of the radiative opacity with respect to density incorrect.

After you’ve make this change, in kap do

./mk
./export

to recompile and export the module.

Step 0: Notice your model has problems

There are many ways in which a MESA model may have problems. It might fail to run with gold tolerances or it might take unexpectedly small timesteps. The timestep is often driven downwards by repeated retries. Information about the retry trigger is printed to the terminal. Or, if the solver requires more iterations than a specified limit (solver_iters_timestep_limit and related controls), it also reduces the timestep and indicates the dt_limit as solver iters.

Note

The debugging steps that follow are not generally applicable when the solver is performing well. For other kinds of bugs (e.g., an option doesn’t produce the expected behavior) one often has clearer idea where the problem may be and so you can head straight to that part of the code.

We will use the 1.3M_ms_high_Z test suite case to illustrate the effect of the bug that we introduced.

In r14460, this case took 377 steps, required 0 retries, and there were a total of 1850 iterations of the newton solver. With the bug, this test case stops when it reaches max_model_number after 500 steps, having done 12 retries and a total of 8145 newton iterations.

Navigate to this test (in star/test_suite) and run it. In the terminal output, you can see that the timesteps are often set by solver iters.

       step    lg_Tmax     Teff     lg_LH      lg_Lnuc     Mass       H_rich     H_cntr     N_cntr     Y_surf   eta_cntr   zones  retry
   lg_dt_yr    lg_Tcntr    lg_R     lg_L3a     lg_Lneu     lg_Mdot    He_core    He_cntr    O_cntr     Z_surf   gam_cntr   iters
     age_yr    lg_Dcntr    lg_L     lg_LZ      lg_Lphoto   lg_Dsurf   C_core     C_cntr     Ne_cntr    Z_cntr   v_div_cs       dt_limit
__________________________________________________________________________________________________________________________________________________

         10   7.186162   6026.290   0.310381   0.310381   1.300000   1.300000   0.639691   0.003250   0.320000  -2.349937    805      0
   4.820702   7.186162   0.119685 -44.790170  -1.003419 -99.000000   0.000000   0.320132   0.018721   0.040000   0.071396     16
 8.4252E+05   1.739919   0.314263 -14.875409 -99.000000  -7.019694   0.000000   0.005826   0.004199   0.040176  0.000E+00  solver iters
                               rel_E_err    5.8461078509513597D-13
                       log_rel_run_E_err      -11.0597023512447645

         13   7.185990   6017.566   0.300802   0.300802   1.300000   1.300000   0.639670   0.003358   0.320000  -2.355541    805      0
   4.873378   7.185990   0.113850 -44.803173  -1.013273 -99.000000   0.000000   0.320139   0.018721   0.040000   0.071285     15
 1.0438E+06   1.737310   0.300077 -14.750470 -99.000000  -7.010156   0.000000   0.005733   0.004199   0.040192  0.000E+00    varcontrol
                               rel_E_err    7.3563886572871659D-13
                       log_rel_run_E_err      -10.9707092147706380

save LOGS/profile2.data for model 13
         20   7.185235   5994.060   0.274258   0.274258   1.300000   1.300000   0.639610   0.003646   0.320000  -2.362919    806      0
   4.851161   7.185235   0.101141 -44.848473  -1.043016 -99.000000   0.000000   0.320157   0.018721   0.040000   0.071188     18
 1.5910E+06   1.733084   0.267860 -15.352530 -99.000000  -6.988274   0.000000   0.005486   0.004199   0.040233  0.000E+00  solver iters
                               rel_E_err    5.1727204553814077D-13
                       log_rel_run_E_err      -10.1566890117973099

Step 1: Activate debugging options

There are standard inlist commands available that can reveal more about the struggle that the solver is going through. They are stored in the aptly-named debugging_stuff_for_inlists which lives in star/test_suite directory. The full list of commands there is

      
      !photo_interval = 1
      !profile_interval = 1
      !history_interval = 1
      !terminal_interval = 1

! FOR DEBUGGING

      !report_solver_progress = .true. ! set true to see info about solver iterations
      !report_ierr = .true. ! if true, produce terminal output when have some internal error
      !stop_for_bad_nums = .true.
      !trace_evolve = .true.
      !fill_arrays_with_NaNs = .true.

      !solver_save_photo_call_number = 0
         ! Saves a photo when solver_call_number = solver_save_photo_call_number - 1
         ! e.g., useful for testing partials to set solver_call_number = solver_test_partials_call_number - 1

      !solver_test_partials_call_number = 4142
      !solver_test_partials_k = 2616
      !solver_test_partials_iter_number = 4
      !solver_test_partials_dx_0 = 1d-6
      !solver_test_partials_var_name = 'o16' ! 'all' or 'lnd', 'lnT', 'lnR', 'L', 'v', etc.    '' means code sets 
      !solver_test_partials_equ_name = 'lnP' ! 'all' or 'dlnE_dt', 'dlnd_dt', 'dlnR_dt', 'equL', etc   '' means code sets 
      !solver_test_partials_sink_name = 'si28' ! iso name to use for "sink" to keep sum = 1
      !solver_test_partials_show_dx_var_name = 'h1'

         ! equ name can also be one of these
            ! 'lnE', 'lnP', 'grad_ad' to test eos
            ! 'eps_nuc' to test net
            ! 'non_nuc_neu' to test neu
            ! 'gradT', 'mlt_vc' to test mlt
            ! 'opacity' to test kap

      !solver_test_partials_write_eos_call_info = .true.
      
      !solver_test_partials_k_low = -1
      !solver_test_partials_k_high = -1
      
      !solver_test_eos_partials = .true.
      !solver_test_kap_partials = .true.
      !solver_test_net_partials = .true.
      !solver_test_atm_partials = .true.

      !report_all_dt_limits = .true.
      !report_solver_dt_info = .true.
      
      !show_mesh_changes = .true.
      !mesh_dump_call_number = 5189
      !okay_to_remesh = .false.
      
      !energy_conservation_dump_model_number = -1
      
      !use_DGESVX_in_bcyclic = .true.
      !use_equilibration_in_DGESVX = .true.
      !report_min_rcond_from_DGESXV = .true.

      ! solver debugging
      !solver_check_everything = .true.
      
      !solver_epsder_struct = 1d-6
      !solver_epsder_chem = 1d-6
      
      !report_solver_dt_info = .true.
      !report_dX_nuc_drop_dt_limits = .true.
      !report_bad_negative_xa = .true.

Copy the contents of debugging_stuff_for_inlists to the controls section of your inlist. Note that many test suite inlists already contain these options.

Begin by uncommenting the following lines to get info about the newton solver progress:

report_solver_progress = .true. ! set true to see info about newton iterations
report_ierr = .true. ! if true, produce terminal output when have some internal error

Step 2: Run the model and find the bad spot

Running the model again will now produce a lot more output, giving information about each and every call to the solver. That output will look something like this:

                                 solver_call_number          17
                              gold tolerances level           2
                   correction tolerances: norm, max    3.0000000000000001D-05    3.0000000000000001D-03
                tol1 residual tolerances: norm, max    9.9999999999999994D-12    1.0000000000000001D-09
17    1  coeff  1.0000     avg resid  0.518E-05      max resid dv_dt     1  0.111E-01  mix type xx000     avg corr  0.474E-03    max corr lnd   556  0.710E-02  mix type 00000     avg+max corr+resid
17    2  coeff  1.0000     avg resid  0.561E-06      max resid dv_dt     1  0.275E-02  mix type xx000     avg corr  0.140E-03    max corr lnd   493  0.140E-02  mix type 11100     avg corr, avg+max resid
17    3  coeff  1.0000     avg resid  0.384E-06      max resid dv_dt     1  0.521E-02  mix type xx000     avg corr  0.262E-04    max corr lnd   494  0.253E-03  mix type 11000     avg+max resid
17    4  coeff  1.0000     avg resid  0.681E-06      max resid dv_dt     1  0.819E-02  mix type xx000     avg corr  0.560E-04    max corr lnd   494 -0.555E-03  mix type 11000     avg corr, avg+max resid
                tol2 residual tolerances: norm, max    1.0000000000000000D-08    1.0000000000000001D-05
17    5  coeff  1.0000     avg resid  0.104E-05      max resid dv_dt     1  0.121E-01  mix type xx000     avg corr  0.780E-04    max corr lnd   494  0.758E-03  mix type 11000     avg corr, avg+max resid
17    6  coeff  1.0000     avg resid  0.159E-05      max resid dv_dt     1  0.177E-01  mix type xx000     avg corr  0.113E-03    max corr lnd   494 -0.110E-02  mix type 11000     avg corr, avg+max resid
17    7  coeff  1.0000     avg resid  0.251E-05      max resid dv_dt     1  0.263E-01  mix type xx000     avg corr  0.166E-03    max corr lnd   494  0.161E-02  mix type 11000     avg corr, avg+max resid
17    8  coeff  1.0000     avg resid  0.403E-05      max resid dv_dt     1  0.383E-01  mix type xx000     avg corr  0.246E-03    max corr lnd   494 -0.238E-02  mix type 11000     avg corr, avg+max resid
17    9  coeff  0.5000     avg resid  0.185E-05      max resid dv_dt     1  0.936E-02  mix type xx000     avg corr  0.359E-03    max corr lnd   494  0.348E-02  mix type 11000     avg+max corr+resid
17   10  coeff  0.5000     avg resid  0.777E-06      max resid dv_dt     1  0.227E-02  mix type xx000     avg corr  0.857E-04    max corr lnd   494 -0.827E-03  mix type 11000     avg corr, avg+max resid
17   11  coeff  1.0000     avg resid  0.262E-06      max resid dv_dt     1  0.329E-02  mix type xx000     avg corr  0.226E-04    max corr lnd   494  0.220E-03  mix type 11000     avg+max resid
17   12  coeff  0.5000     avg resid  0.660E-07      max resid dv_dt     1  0.754E-03  mix type xx000     avg corr  0.306E-04    max corr lnd   494 -0.296E-03  mix type 11000     avg corr, avg+max resid
17   13  coeff  0.5000     avg resid  0.190E-07      max resid dv_dt     1  0.185E-03  mix type xx000     avg corr  0.712E-05    max corr lnd   494  0.691E-04  mix type 11000     avg+max resid
17   14  coeff  0.5000     avg resid  0.540E-08      max resid dv_dt     1  0.409E-04  mix type xx000     avg corr  0.172E-05    max corr lnd   494 -0.166E-04  mix type 11000     avg+max resid
17   15  coeff  0.5000     avg resid  0.200E-08      max resid dv_dt     1  0.106E-04  mix type xx000     avg corr  0.382E-06    max corr lnd   494  0.369E-05  mix type 11000     max resid
17   16  coeff  0.5000     avg resid  0.754E-09      max resid dv_dt     1  0.216E-05  mix type xx000     avg corr  0.103E-06    max corr lnd   494 -0.100E-05  mix type 11000     okay!

The first line gives solver_call_number, which uniquely identifies a particular solver call. (It will differ from the model number after retries have occurred.) Then, there is an indication of what set of solver tolerances are being used and what those tolerances are. After that, there is a series of lines that gives iteration-by-iteration summaries of what the solver did. The iteration number is the second column. After certain numbers of iterations (user-defined, see e.g., gold_iter_for_resid_tol2), the solver relaxes the residual tolerances and the new tolerances are indicated.

Other columns show information about the residuals and corrections. The “residual” is the left over difference between the left and right hand sides of the equation we are trying to solve. We do iterations to reduce that, but we are limited by the non-linearity of the problem and the quality of the estimates for the derivatives. The “correction” is the change in the primary variable, calculated using Newton’s method, so the Jacobian and residuals give a correction that would make the next residual vanish if the problem were linear and the Jacobian were exact, neither of which are true. So the best we can hope for is that the corrections will get smaller next iteration. When the maximum residual/correction is shown, the equation/variable and its cell index are indicated. The mixing types for the cells around these locations are also indicated. The 1’s are the convection and the 0’s are for no-mixing. (The full list of mixing types is in const_def.)

The last column is the simplest to understand. It tells us that for the early iterations, the solution was rejected because most everything was unsatisfactory. Then, as the iterations proceed and the residuals and corrections shrink, eventually only the maximum residual is unsatisfactory. Finally, the solver decides the solution is good enough.

Note

Less artificial bugs may behave somewhat differently than this toy problem. In this example, the solver is pretty much always struggling, so you could pick almost any step as the bad one. In practice, models often only experience problems under specific conditions and so a much smaller subset of steps are bad. Generally, you want to pick a step just as things start to go wrong. In this example, the solver also eventually succeeded. In practice, one often see cases where the newton iterations will completely fail to drive the max residual below the tolerances and the solver will give up, forcing a retry and reducing the timestep.

Step 3: Check the partial derivatives

From the terminal output in the previous part, we can see that the max residual of dv_dt equation is difficult to reduce and is located in cell 1.

We want to find the source of this bad residual by checking correctness of the partial derivatives of the corresponding equation. MESA can compare the analytic derivative that it has been given with a numerical estimate of the derivative based on finite differences and Ridder’s method of polynomial extrapolation (aka dfridr). To do so, we must ask for the derivatives at a particular cell at a particular iteration number of a particular solver call. This can be achieved with the inlist options:

solver_test_partials_call_number = 17
solver_save_photo_call_number = 17
solver_test_partials_iter_number = 8
solver_test_partials_k = 1
solver_test_partials_equ_name = 'dv_dt'
solver_test_partials_var_name = 'all'
solver_test_partials_dx_0 = 1d-6

Setting solver_save_photo_call_number = 17 produces a photo at the preceding step (x016) and so re-running the bad step becomes as easy as doing ./re. Debugging is itself an iterative process, so you want to make sure you can rapidly and consistently reproduce the problem. A little time invested up front letting you easily and rapidly trigger the problem pays dividends when you have repeat the procedure a dozen times.

Now when you do ./rn (or ./re after the photo exists) and the model will stop after the specified solver call and you’ll see

                                         equ name dv_dt
*****                 log dfridr rel_diff partials wrt  lnd(k)   of dv_dt(k)      1     0.677   log uncertainty    -9.668   analytic   -2.9597633549276736E-01   numeric   -1.7033292388187307E+00
                      log dfridr rel_diff partials wrt  lnd(k+1) of dv_dt(k)      1   -99.000   log uncertainty   -99.000   analytic    0.0000000000000000E+00   numeric    0.0000000000000000E+00

                      log dfridr rel_diff partials wrt  lnT(k)   of dv_dt(k)      1   -10.058   log uncertainty   -10.644   analytic   -6.9708999322145324E+00   numeric   -6.9708999316051248E+00
                      log dfridr rel_diff partials wrt  lnT(k+1) of dv_dt(k)      1   -99.000   log uncertainty   -99.000   analytic    0.0000000000000000E+00   numeric    0.0000000000000000E+00

?????                   log dfridr rel_diff partials wrt  L(k)   of dv_dt(k)      1    -4.255   log uncertainty    -4.272   analytic    6.3131382934316702E-39   numeric    6.3127875526588140E-39
                        log dfridr rel_diff partials wrt  L(k+1) of dv_dt(k)      1   -99.000   log uncertainty   -99.000   analytic    0.0000000000000000E+00   numeric    0.0000000000000000E+00

                      log dfridr rel_diff partials wrt  lnR(k)   of dv_dt(k)      1   -11.110   log uncertainty   -10.064   analytic   -2.0080506394821409E+00   numeric   -2.0080506394665569E+00
                      log dfridr rel_diff partials wrt  lnR(k+1) of dv_dt(k)      1   -99.000   log uncertainty   -99.000   analytic    0.0000000000000000E+00   numeric    0.0000000000000000E+00

This output shows the derivatives in the cell for each equation with respect to each variable. The column after the cell number shows the relative difference between the numerical (dfridr) and analytic derivatives. When things are working well, the relative differences go down to < 1d-12, but don’t panic if you see 1d-7. That can still work. But 1d-4 or more is a bad sign. Here, our bad derivative jumps out with an order unity relative error and is highlighted by the **** on the left side. So we now know the bad derivative is the one with respect to density.

The relative uncertainty in the numerical estimate of the derivative is also indicated. When this is comparable to the relative error between the numerical and analytic derivatives, then you’re probably not learning anything about the quality of the analytic derivative. (See for example the line marked with ???? above.) Sometimes that sort of result can be a false alarm caused by the particular choice of starting difference for the dfridr search. This is set by solver_test_partials_dx_0 and is typically something around 1d-6. Usually that works, but if you get a large err that makes no sense, you should try both larger and smaller values of to see if the dfridr result changes.

It is often the case that bad derivatives in an equation come from bad derivatives in the microphysical inputs that enter into the equation. Since this is common, MESA supports easily checking the basic eos, kap, and net quantities. In particular, the value of solver_test_partials_equ_name can be set to 'lnE' or 'lnP' (testing eos), to 'eps_nuc' (testing net), or 'opacity' (testing kap).

Apply your knowledge about which equation showed the issue. If the problem were in the energy equation, you would probably want to test lnE and eps_nuc. Since this issue appeared in the momentum equation, we will want to test lnP and since it was in the surface cell, we will want to test opacity as that enters into the atmosphere boundary condition.

We already found the issue was in the density derivatives, so we can now restrict our attention to that derivative by setting solver_test_partials_var_name = 'lnd'.

Setting solver_test_partials_equ_name == 'lnP' and restarting shows no problem

log dfridr rel_diff partials wrt  lnd(k)   of lnP(k)      1    -9.791   log uncertainty   -12.139   analytic    9.9965278727210283E-01   numeric    9.9965278743368469E-01
log dfridr rel_diff partials wrt  lnd(k+1) of lnP(k)      1   -99.000   log uncertainty   -99.000   analytic    0.0000000000000000E+00   numeric    0.0000000000000000E+00

but solver_test_partials_equ_name == 'opacity' and restarting shows

*****               log dfridr rel_diff partials wrt  lnd(k)   of opacity(k)      1     0.301   log uncertainty    -9.108   analytic   -2.9110358120328123E-01   numeric    2.9110358102038758E-01
                    log dfridr rel_diff partials wrt  lnd(k+1) of opacity(k)      1   -99.000   log uncertainty   -99.000   analytic    0.0000000000000000E+00   numeric    0.0000000000000000E+00

Comparing the bad analytic derivative to the numerical one reveals the sign error that we introduced.

We’ve reached the end of the part of debugging that can be done with inlists only. We will need to edit the code in order to MESA to dive deeper into a module or to check individual terms in the equation derivatives.

As a test case maintainer, this may be the point at which you stop, report the issue, and turn things over to others. If the issue is in the eos, you might dig in a bit more to see which component of the eos is active by setting solver_test_partials_write_eos_call_info = .true. and include that info in your report.

Step 5: Descend into an individual module

If the bug has already been found, this step isn’t necessary, but often one must continue deeper into a module. For the modules atm, eos, kap, net, and rates, it is possible to continue to do some of this debugging from star. This is extremely helpful because it means that the precise properties of the troublesome cell from star are passed into the module.

In the inlist set:

solver_test_kap_partials = .true.

(Note that using this option automatically sets MESA to run in single-thread mode.)

Since will specify which quantity to check in the code, we should stop selecting the equation in the inlist, but can continue to specify which derivative of that quantity to check:

solver_test_partials_equ_name = ''
solver_test_partials_var_name = 'lnd'

The variables kap_test_partials_val and kap_test_partials_dval_dx can now be set within kap and thereby communicate their values into star and dfridr.

At the end of Get_kap_results in kap/private/kap_eval.f90 — near where we introduced our bug — add

kap_test_partials_val = log(kap_rad)
kap_test_partials_dval_dx = dlnkap_rad_dlnRho

Again, the key point of this step is that we’re passing up a value (kap_rad) that we wouldn’t normally have access to from star. This procedure allows you to debug deep within a module from star.

We changed code in kap, so we need to do ./mk && ./export in star and then in the work directory also recompile before we restart ./mk && ./re.

The output should clearly illustrate that this derivative has the wrong sign. Before making a change that corrects the value of dlnkap_rad_dlnRho, consider putting what you think the right answer is in kap_test_partials_dval_dx and re-running to check that your new expression is OK. (If you first fix the bad derivative at the location of the bug, that change will mean that the newton iterations won’t be the same any more and you may not hit the same conditions and or trigger dfridr.)

After you’ve checked, make the bug fix, turn off the debugging options, run your model, and hopefully see improved performance.

Advanced: Investigate the bad cell and equation in detail

Note

We will now pretend we didn’t find the bug and re-use this setup to illustrate how to dig in at an even deeper level.

MESA equations and structure variables have integer indices. The full list is in star_data/public/star_data.inc, but the most common ones are

! index definitions for the variables (= 0 if variable not in use)
   integer :: i_lnd ! ln(cell density average by mass)
   integer :: i_lnT ! ln cell temperature average by mass
   integer :: i_lnR ! ln radius at outer face of cell
   integer :: i_lum ! luminosity at outer face of cell
   integer :: i_v ! Lagrangian velocity at outer face of cell
   integer :: i_u ! Lagrangian velocity at center of cell
   integer :: i_lnPgas ! ln cell gas pressure average by mass

! index definitions for the equations (= 0 if equation not in use)
   integer :: i_dlnR_dt ! dlnR/dt = v/r
   integer :: i_dv_dt ! momentum conservation using v
   integer :: i_du_dt ! momentum conservation using u
   integer :: i_dlnd_dt ! mass conservation
   integer :: i_dlnE_dt ! energy conservation
   integer :: i_equL ! luminosity

In Step 3, we found that we want to investigate the lnd derivative of the dv_dt equation in cell k = 1. If we hadn’t been able to so easily narrow things down to kap, then we would need to make a more detailed investigation of the equation in this cell.

To do so, go to star/private/hydro_eqns.f90 and follow the flow subroutine eval_equ_for_solver until you reach the equation that you want investigate.

The surface cell (k = 1) is special and so we see that the dv_dt equation for this cell is provided by PT_eqns_surf. Go to that subroutine. Continue to follow the logic until you get the place that defines the residual of the desired equation. In this case, that is yet one more level down in set_Psurf_BC.

There you’ll see the code that indicates that we’ve reached the dv_dt equation in cell 1.

if (s% u_flag) then
   i_eqn = i_du_dt
else
   i_eqn = i_dv_dt
end if

equ(i_eqn,1) = lnP_bc - s% lnP(1)

This is an extremely simple equation. By driving the residual to zero, MESA requires that the surface cell pressure match the imposed boundary condition.

Now, we want to tell MESA to test the partial derivatives. Near the start of the routine, set

test_partials = (s% solver_iter == s% solver_test_partials_iter_number)
!test_partials = .false.

Almost all the equation routines will already have test_partials code like this. And once you reach one routine that has test_partials, most of the routines it calls are usually also set up to check partials. (But if you are in uncharted territory you might have to add them yourself.)

Note

Since PT_eqns_surf and set_Psurf_BC are only called for k=1, we do not test for zone number. In other equations that are evaluated for multiple k, the analogous test will usually look like

test_partials = (k == s% solve_test_partials_k .and. s% solver_iter == s% solver_test_partials_iter_number)
!test_partials = .false.

Set solver_test_partials_val to a variable that is used in calculating the equation residual. Let’s begin with the residual itself, which should reproduce the bad entry from the previous step in this tutorial.

if (test_partials) s% solver_test_partials_val = equ(i_eqn,1)

Find the call on e00 for i_lnd to find the relevant partial with respect to density.

call e00(s, xscale, i_eqn, i_lnd, 1, nvar, dlnP_bc_dlnd - s% chiRho_for_partials(1))

Note

Recall the MESA Jacobian has a block tridiagonal structure (Fig. 47 in MESA II). The e00 idiom refers to forming the block corresponding to the derivatives of the equations in cell k with respect to the variables in cell k. Similarly, em1 and ep1 refer to the the derivatives of the equations in cell k with respect to the variables in cell k-1 and in cell k+1 respectively. The 3rd and 4th arguments to these subroutines indicate that this call is setting the derivative of a specific equation (recall i_eqn = i_dv_dt earlier) with respect to a specific variable (here i_lnd).

Set solver_test_partials_var to the thing you want to take a derivative with respect to and solver_test_partials_dval_dx to the analytic derivative.

if (test_partials) then
   s% solver_test_partials_var = i_lnd
   s% solver_test_partials_dval_dx = dlnP_bc_dlnd - s% chiRho_for_partials(1)
   write(*,*) 'set_Psurf_BC', s% solver_test_partials_var
end if

We changed code in star, so we need to do ./mk && ./export in star and then in the work directory also recompile before we restart ./mk && ./re. We will repeat this recompilation/restart process many times, so do it in a way that makes it easy for you do over and over.

Since we are now specifying which derivative to check in the code, we must also stop selecting the equations/variables in the inlist.

solver_test_partials_equ_name = ''
solver_test_partials_var_name = ''

solver_test_kap_partials = .false.

After that edit, once you recompile and restart you should see

    set_Psurf_BC           1


                  analytic and numeric partials wrt lnd   -2.9597633549276736D-01   -1.7033292388187307D+00
    log dfridr relative uncertainty for numeric partial   -9.6678068357602811D+00
                                               rel_diff    4.7549507665296238D+00

STOP done solver_test_partials

Again, this looks bad! In the final line, we can see that the analytic estimate of the derivative (what goes in the Jacobian) is order-unity different from the numerically evaluated derivative.

Now that we’ve confirmed which derivative is the problem, we must look for which term or terms are responsible. There are only two in this equation, lnP_bc - s% lnP(1), so you can test them one by one

s% solver_test_partials_val = lnP_bc

and

s% solver_test_partials_var = i_lnd
s% solver_test_partials_dval_dx = dlnP_bc_dlnd

Recompile and restart to confirm that this is the bad term.

You can repeat this with the other term to confirm that lnP(1) and its derivative s% chiRho_for_partials(1) are OK. If they weren’t, then it would be off to the EOS to figure out why. We’ll discuss how to dive deeply into a specific module in the next step.

Once you locate the bad term, find where that term is defined, so that you can investigate what is causing it to be in error. On our way down, we saw that dlnP_bc and its derivatives are set in PT_eqns_surf. So now turn off the partial tests in set_Psurf_BC, switch to PT_eqns_surf, turn on the partial tests there.

Note

When you finish testing a partial in one place, don’t forget to turn off the test in that place before moving on to the new place. Typically the code that reports results by setting solver_test_partials_val, solver_test_partials_var, and solver_test_partials_dval_dx, will also have a write that says who it is. This can be a lifesaver if/when you do forget to turn off one routine before turning on another. The terminal output will show the problem and tell you where it is.

We can start by testing the partial of lnP_bc to confirm our previous finding. (Sample code omitted so you can practice setting the variables yourself.) That should give

   set_Psurf_BC           1

                  analytic and numeric partials wrt lnd    7.0157923462638694D-01   -7.0157923451592796D-01
        dfridr relative uncertainty for numeric partial   -1.6481349835630704D-12
                                               rel_diff    2.0000000001574434D+00

STOP done solver_test_partials

and at this point you can clearly see the sign error we introduced. With that confirmed, we want to start dissecting

dlnP_bc_dlnd = dlnP_bc_dlnPsurf*dlnPsurf_dlnkap*dlnkap_dlnd

For more complex expressions involving many terms, you might need to test term by term.

Note

The dfridr machinery in MESA tests the derivatives with respect to structure variables, so it can’t magically check every derivative. To check dlnP_bc_dlnPsurf you can look at the expression relating lnP_bc and lnP_surf in this routine and confirm it with pencil and paper. To check dlnPsurf_dlnkap you would have to follow the code down into atm. Once there, you could check by hand or set up your own dfridr code that passes in varying values of kap and exercises the module independent of star.

Here, since there’s only one dlnd derivative, dlnkap_dlnd is the clear place to start (even if we didn’t already know it was the issue). Set

s% solver_test_partials_val = log(s% opacity(1))

s% solver_test_partials_var = s% i_lnd
s% solver_test_partials_dval_dx = dlnkap_dlnd

and dfridr should confirm that the bad derivative is the density derivative of the opacity.

The surface cell momentum equation is one of the simpler equations. Two other types of derivatives often arise: derivatives with respect to quantities in neighboring cells and derivatives with respect to composition. The energy conservation equation (get1_energy_equation in hydro_equ_l.f90) provides a good example of an equation with both these features.

When testing a derivative like d_esum_dlnTm1, that is a partial of a quantity esum(k) with respect to lnT(k-1), note that the dfridr code perturbs the structure of cell k = solver_test_partials_k. That is, it can test the partials of equations in cell k-1, k, and k+1 with respect to the structure variables in cell k. (In practice if there is a problem with partials of a quantity in cell k then there are also problems in cells k+1 and k-1. But if you really need to test one of these derivatives in a specific cell, you can always adjust solver_test_partials_k in the inlist.)

Therefore, to check d_esum_dlnTm1 one can do

test_partials = (k-1 == s% solver_test_partials_k .and. s% solver_iter == s% solver_test_partials_iter_number)
!test_partials = .false.

and

s% solver_test_partials_val = esum
s% solver_test_partials_var = s% i_lnT
s% solver_test_partials_dval_dx = d_esum_dlnTm1

This will check the derivative of esum in cell solver_test_partials_k+1 with respect to lnT in cell solver_test_partials_k.

Checking composition derivatives requires setting a sink isotope such that the dfridr perturbations to the chosen isotope are offset by changes in a “sink” isotope to keep the mass fraction sum = 1. The sink needs to be inert in the current situation and have enough abundance to absorb the changes.

When debugging from via the inlist, this isotope can be specified using the control solver_test_partials_sink_name. So for example, addition to setting the usual k, call_number, iter_number, and dx_0, to check partials of the energy equation with respect he4, you might set the following

solver_test_partials_equ_name = 'dlnE_dt'
solver_test_partials_var_name = 'he4'
solver_test_partials_sink_name = 'fe56'

If solver_test_partials_dx_0 is too large, you’ll get an error if the mg24 mass fraction is too small to deal with the desired increase in the targeted isotope.

When you need to check composition derivatives of specific terms, you must pick the various test_partials values in the source code. This can be tricky in that in some places the abundance is given as an index in xa and in others as a variable number for the solver. This offset is s% nvar_hydro, that is the index of the first chem equation is s% nvar_hydro + 1.

If you set solver_test_partials_write_eos_call_info = .true., the output includes also includes composition information like the following so it is easy to find the needed numbers.

xa(j,k) neut           1           6        3312    1.0010960558047912D-99
  xa(j,k) h1           2           7        3312    0.0000000000000000D+00
xa(j,k) prot           3           8        3312    1.0010960558047908D-99
 xa(j,k) he3           4           9        3312    0.0000000000000000D+00
 xa(j,k) he4           5          10        3312    6.7423592032960378D-01
 xa(j,k) c12           6          11        3312    1.0630746138489515D-01
 xa(j,k) n14           7          12        3312    6.1988280539418319D-03
 xa(j,k) o16           8          13        3312    2.0390257320339500D-01
xa(j,k) ne20           9          14        3312    5.7660956389211995D-03
...
xa(j,k) fe56          20          25        3312    1.4153093511237100D-03

So then the in the code you would do something like

if (test_partials) then
   s% solver_test_partials_dx_sink = 20 ! fe56 (index in xa)
   s% solver_test_partials_var = 10 ! he4 (variable number)
   s% solver_test_partials_dval_dx = s% d_epsnuc_dx(5,k) ! he4 (index in xa)
   write(*,*) 'get1_energy_eqn', s% solver_test_partials_var
end if

Diagnosing Meshing Problems

Note

This was the file DEBUG_mesh.notes. It needs modernized.

to get info about why the mesh is doing what it is doing,

in controls section of inlist:

show_mesh_changes = .true.

run to get current mesh call number, n

in controls section of inlist:

mesh_dump_call_number = n

run to get plot data

mesh_plan.rb shows input data that was used to form the plan

mesh.rb shows new mesh (with some old values interpolated to the new mesh for comparison)

Floating point exceptions

Floating point exceptions (fpe) occur when the code attempts to perform an illegal math operation, for instance x/0, sqrt(-1), or log(0). In these cases any number that uses the result of one of these expressions is also undefined.

To test MESA for floating point exceptions we can turn on the compiler checks that will flag when an issue occurs:

Set the environment variable

MESA_FPE_CHECKS_ON=1

Then run ./clean and ./install in MESA_DIR.

This setting adds the following options in utils/makefile_header

   FCbasic += $(FCtrapNANs) -fbounds-check -finit-derived -Wuninitialized -Warray-bounds -fstack-protector-all -D_FORTIFY_SOURCE=2
   FCopt = -O0 -ftree-vectorize 
   SKIP_NAN_TRAPS = NO

and acts as if you set the controls inlist option

fill_arrays_with_nans = .true.

to make sure you catch any uninitialized array accesses.

Step -1: Introduce a bug (again)

Lets add a fpe in a test case to see what happens:

Alter star/test_suite/1.3M_ms_high_Z/src/run_star_extras.f90 and in extras_finish_step add:

write(*,*) log(-1d0*s% model_number)

Step 0: Notice your code has a problem

Run ./mk and ./rn to run the test case and you’ll get the following output:

Program received signal SIGFPE: Floating-point exception - erroneous arithmetic operation.

Backtrace for this error:
#0  0x7efca94096af in ???
#1  0x7efca9cc6330 in ???
#2  0x405f10 in __run_star_extras_MOD_extras_finish_step
      at ../src/run_star_extras.f90:155
#3  0x425372 in __run_star_support_MOD_after_step_loop
      at ../job/run_star_support.f90:634
#4  0x427d63 in __run_star_support_MOD_run1_star
      at ../job/run_star_support.f90:162
#5  0x407076 in __run_star_MOD_do_run_star
      at ../../../../star/job/run_star.f90:26
#6  0x4070eb in run
      at ../src/run.f90:13
#7  0x407137 in main
      at ../src/run.f90:2
./rn1: line 6: 2166962 Floating point exception(core dumped) ./star

Step 1 Diagnose the issue

The important line is the first line that does not include ???, in this case it is

#2  0x405f10 in __run_star_extras_MOD_extras_finish_step
      at ../src/run_star_extras.f90:155

This is telling us that the problem occurs at line 155 in ../src/run_star_extras.f90

Note

The ../src is because the file path is relative to the make/ folder in your work folder

At this point you should go to that line and work out why the number is undefined.

Uninitialised variables

Uninitialised variables occur when we use a variable before it has been defined, for example

real(dp) :: x,y
y = 2 * x

If you are lucky then x will be zero, if you are unlucky it can be any random number that happened to be in the memory location that x occupies. This can lead to results that change either each time the model runs or gives different values between different machines.

By default MESA compiles with the -finit-real=snan option, this will set all scalar floating point variables to a NaN when they are declared.

Note

This does not protect integers

As the variable will be a NaN we can use the floating point exceptions mechanism to track them down, so follow the instructions for debugging floating point exceptions.

Arrays of floating point numbers

These can also be protected by wrapping the array after its declared with:

call fill_with_NaNs(array)

use utils_lib, only:: fill_with_NaNs
real(dp),dimension(:),allocatable :: x

allocate(x(1:10))
call fill_with_NaNs(x)

Note

Never try to fill the array yourself with a NaN, say by setting each element to 1/0. We want to track down the location where an array element is being used before being set, not the point where we set the element to a NaN. If you set the array yourself the fpe checking will warn you about that location, not where the array is being before being set. fill_with_NaNs does some clever tricks to set each element to the bit pattern that corresponds to a NaN, which does not trigger the fpe checks.

Memory leaks