Are Monte Carlo Simulations Good Predictors of Retirement Outcomes? - January 2023

hi Derek

0:14

I had my uh speaker on mute there for a second if somebody said anything hey everyone

0:23

looks like a lot of people joining

0:31

everyone just a few minutes to get logged in here

1:22

foreign

1:51

the webinar invitation sorry about that folks yeah we'll get that fix for next time

1:58

I'll give everybody just an extra minute because of that in case they had to re-register

2:05

start here in just a few moments thank you for the feedback on that

2:36

okay we'll go ahead and get started uh Welcome to our fourth Tuesday of

2:42

every month webinar today's topic is our Monte Carlo simulations good predictors

2:47

of retirement outcomes with Justin Fitzpatrick and Derek Tharp and this

2:52

will be we will be having a an educational webinar uh the fourth Tuesday of every month and also for our

3:00

users we do have a third Tuesday of every month lab talk Tuesday's webinar

Lab Talk Tuesday

3:05

so feel free to join both of those uh at the at throughout this webinar we will

3:11

have a place on the side where you can add questions and you can also upvote each other's questions if you would like

3:17

them to be addressed at the end we will have some time for questions at the end also at the end of this webinar survey

3:25

will be coming around via Zoom please take a moment to fill that out as well

3:31

because feedback is very important to us and we can always look for new topics or

3:36

address questions and last but not least if you have planned specific questions please always visit us at our help

3:43

center within the app or email us at info income laboratory laboratory.com so

3:49

with that I'm going to pass it over to Justin and we'll get rocking and rolling

3:54

all right thanks Taylor thanks Derek thanks everybody for for joining us so

4:00

um this presentation is um is going over some work that that Derek and I have been doing

4:07

um and and it's really around you know trying to figure out how to do

4:13

retirement planning in the way that is is best for clients and part of that is

4:19

really kind of basic research that you know surprisingly maybe it hasn't been

Are Monte Carlo Simulations Accurate

4:24

done or or you know there's not a lot in this area um

4:30

so when when people use uh tools that make forecasts in order to as as part of

4:37

their retirement planning and for some retirement plans this is super important um they're assuming that the the

4:45

forecasts they get the numbers like probability of success um are reasonably accurate right I mean

4:52

I don't think anybody is expecting that um you know it's it's precise down to the decimal point or anything like that

4:58

but you certainly would expect for example that if you got in a um

5:03

in in a planning tool something like an 80 probability of success

5:09

um that in fact uh you know when as often as that planning tool gives an 80

5:14

probability of success that in fact those those plans tend to succeed about 80 of the time right I mean that's just

5:22

kind of an assumption um we can kind of miss that assumption sometimes because uh you know the tools

5:27

can can be you know quite good presented at a certain way and we just kind of um trust right that that things are done

5:34

well but it's important to remember that anytime we're using analysis to do

5:40

planning for for clients um we're using a model of the world not the world itself right so we're always

5:46

one step at least removed from that um and so you know these because these

5:52

kinds of estimates can be super important for the the the kinds of

5:58

advice that that you give to clients um it's worth asking you know are are the are these accurate uh forecasts

6:05

actually accurate and and you know are there any things that we can learn about these forecasts that could help us

6:12

um do better planning for for clients because you know we're giving really consequential Financial advice to people

6:18

often based on these kinds of numbers now in income lab software you'll never

6:25

see a probability of success gauge you'll never see the words probability of success or probability of failure

6:31

because um you know everything done in income lab is is adjustment focused so it's the

6:38

idea that you're a guide through retirement and then by guiding them through retirement um they wouldn't fail or run out of

6:45

money they would simply adjust their behavior um but even an income lab you know kind

6:50

of under the hood estimates of risk and forecasts and things are important so this is

6:56

um this is just as important for income lab as as for you know kind of generalized uh software that that does

7:03

use success and failure framing um so what we assume when we when we get

7:10

something like a 80 probability of success is that we're somewhere around this middle blue box right now again the

7:16

actual success rate I think we'd all be plenty happy if it was 75 to 85 or something like that right but

7:23

um we want it to be roughly in line what we don't want is that I see an 80 probability of success forecast but in

7:31

fact those kinds of plans tend to succeed only 40 of the time right so

7:36

that would be um a Model A forecasting model that is vastly underestimating risk

7:43

um I actually also don't want a forecasting model that says you know I have a 40 chance of success when in fact

7:51

um those kinds of plans end up succeeding 70 of the time that would be overestimating risk

7:57

um this is probably a little now this example is Extreme so I certainly wouldn't want this this is this is way

8:03

out of whack um but if I had to choose between you know out slightly overestimating or

8:08

slightly underestimating I would probably choose overestimating in kind of the following the philosophy of

8:15

you know kind of under promise over deliver um because you know the news to a client

8:21

that whoops actually things are better than I thought they were um is is uh is is good news whereas

8:26

actually things are worse than I thought they were is is bad news so

Predicting Rain

8:33

um Derek and I have done some work on on trying to figure out you know looking at different approaches to modeling the

8:40

world different approaches to Monte Carlo simulation historical simulation and so on you know how how do these

8:47

things play out in in the real world and I think the best way to understand how you can study that question is to look

8:54

at a different one that we that that seems pretty familiar and pretty straightforward which is

8:59

um forecasting rain so uh you know meteorologists use all

9:05

sorts of different weather models right which could be you know analogous to different kinds of simulation models

9:10

different kinds of Monte Carlo analysis historical analysis and so on so I think the the analogy goes pretty far actually

9:18

and also in predicting rain we're not typically given categorical forecasts

9:25

right yes no will will it rain tomorrow yes or no instead we get probabilistic

9:30

forecasts which is the same thing that that you may be familiar with in retirement planning or financial

9:36

planning more generally so let's say for example that that these

9:42

are two possible forecasts you know will it rain tomorrow yes or well there's a 75 chance it'll rain tomorrow

9:49

now if it doesn't rain tomorrow clearly the probabilistic forecast was more

9:55

accurate right because it didn't it it it it sort of allowed for the possibility that that it wouldn't rain

10:03

um let's say it does rain tomorrow and these were the forecasts it said you know one is no it definitely won't and

10:09

the other one says well there's a 25 chance clearly again the probabilistic forecast is is

10:16

um more accurate in this case because it does allow for the chances that that it will rain it just says that the

10:22

likelihood is low um and so when we want to um

10:28

kind of give a grade to different forecasting models we want to look at

10:33

how far off they are from the actual outcomes um and one way to think about that is

The Briar Score

10:40

um just looking at the error um so for example if if I say yes it

10:47

will definitely rain but it doesn't rain my error is we could say 100 right I'm completely off yeah if I say it will not

10:54

rain zero percent chance but it does rain again I'm I'm 100 off so my error

10:59

is you know 100 um for probabilistic forecasts are generally

11:05

not zero or a hundred percent so for for anything in between it's it's that

11:10

that's still what we're looking at we're saying well how far off from the actual outcome was it it either rained or it

11:15

didn't um and so there's always some error so for example in in this case I'm

11:22

comparing a good rain model and a bad rain model um and they basically are always exactly

11:28

the opposite of each other so the good model in the first case is predicting 90 chance of rain the bad model 10 it does rain

11:37

um and clearly the good model is closer right it's it's it's not as far off as the bad model and there's actually a way

11:44

to to sum up and average these these scores um so for the for the math nerds Among

11:50

Us um this is just the the mean squared error so it's always positive

11:56

um and I want my error to be low um so uh this is actually called The

12:01

Briar score for those uh who who love stats um but you can just think of it as the higher the number

12:07

um the worse it is and there's lots of interesting things to talk about um around these errors and the kinds of

12:13

errors that we we have and so on but the thing to remember here is we can score a forecasting model in this case a rain

12:20

forecasting model um by the error and all we would need is lots and lots of examples of the

12:28

forecasts that were made ahead of time and then the actual outcome for the thing it was forecasting

12:34

um and then we can we can come up with these errors you can also look at

Rain Forecasts

12:39

um forecasts a little more granularly to see if there are any Trends or patterns

12:45

places where the rain forecasting model in this case you know does particularly well and particularly poorly or does it

12:51

have a particular bias in One Direction or another and the way we do that is we would just

12:57

group together all of the times that the the forecasting model let's say

13:04

um predicted an 80 chance of rain and then see how often it actually rained during that time

13:10

do the same for 70 and 60 and 50 and really every everything there right so

13:15

now obviously in this case we would probably want you know several years worth of data we'd probably want um

13:21

forecasts in lots of different places right so maybe we we would have

13:26

um you know some in Tucson Arizona and some in Tampa Florida and right so we

13:33

would have kind of a lot of different environments for this model to to test and I think that's somewhat equivalent

13:40

to you know different kinds of retirement income plans right some that

13:46

are really trying to stretch and and and and produce a lot of income for a family given the resources some that are a lot

13:53

more conservative and really are it's a more frugal family given their resources and and so on

13:58

but once we do that we can see um different kinds of biases so in this

14:04

case these are just you know made up examples let's imagine that there's a a green model and a red model and you can

14:11

see the green model is generally predicting

14:16

um more rain sorry uh yeah did I get this I might

14:22

have put this one wrong um no this is this is I think done correct

14:32

sorry the green model actually it's a dry bias here so this is predicting um uh more less rain than they actually get

14:39

and the other one is predicting more rain than we actually get now the

14:46

perfect calibration of a model would be that you know right along the uh

14:53

the the the the diagonal right every time I predict an 80 if I if I look at my bucket of 80 predictions eighty

15:00

percent of in any in 80 of those times it actually rained right and again you're not likely to see

15:07

um Kind of Perfect calibration in a model of something complex so

15:12

um even in rain I would find that that um you know pretty pretty hard to to

15:18

find um you know you can you can look at like if you're uh into stats you may have looked at like the 538 Blog and you know

15:25

look at their predictions on um major league baseball games for example

15:31

um you you won't see great calibration there are other things that that can be very well calibrated though

Calibrations

15:37

um for retirement income plans um this is

15:43

uh this is one of those places where because the world is so complex I I

15:50

really going into this did not expect to see um particularly good calibrations or

15:56

particularly low errors these are the kinds of things that you know modeling of them is the model of the world is

16:02

likely not to capture everything about the world and and do a really good job

16:08

um however um we we do expect there to be differences across models so so Derek

16:13

and I looked at four different ways to um to produce

16:19

uh probabilistic forecasts right so this would be you know three different kinds of Monte Carlo and historical analysis

16:27

and again we're really focusing on even though this isn't how income lab does

16:32

things or presents things to clients we're just taking a very simple question here which is predict probability of

16:38

success and then see whether a plan succeeded or failed

16:43

so we looked at traditional Monte Carlo um reduced Capital Market assumption Monte Carlo so

16:50

this is just uh well I'll show you in a second what each of these are regime based Monte Carlo and historical

Capital Market Assumption

16:56

analysis um and in order to test um the actual outcomes of particular

17:03

forecasts what we had to do is set up a a a a very just systematic formulaic way

17:11

to produce Capital Market assumptions um and so the way we did that is for

17:18

traditional Monte Carlo which most people have used um we produced one set of Capital Market

17:24

assumptions using the averages from the preceding 30

17:30

years so if I was making a prediction um in I don't know January of 1980 I

17:37

would use the 30 years preceding that to create my Capital Market assumptions for reduced Capital Market assumption uh

17:46

Monte Carlo we just reduced the ones from traditional by two percent right so this was meant to sort of reflect

17:53

um what some advisors and firms do which is say hey you know I think using

17:59

historical averages although you know clearly that's there is some motivation behind that I think it let's be a little

18:06

more conservative and bump down our our Capital Market assumptions

18:11

for regime-based Monte Carlo we also use historical averages so again

18:17

if I'm in January of 1980 I'm only looking to the Past

18:22

um but here we allowed um the the forecaster to use all of history but they

18:29

um they first filtered history only to use the the times in the past that were

18:36

closest to that point um and the filter we used was was Cape so um cyclically adjusted p e ratio and

18:44

we just got rid of half of History the half of history that was that had the the cape that was farthest away from

18:50

where we were in January of 1980 for example um and that's how we created the uh the

18:56

regime-based Monte Carlo for historical again January of 1980 we

19:02

just allowed that um to use the return and inflation sequences that were available up to that point in time right

19:08

so there's never any you know foresight allowed this is all kind of we're trying to make sure that at each point in

19:14

history we're acting just like we do today um where we can't see the future

Capital Market Assumptions

19:19

and just a reminder for um maybe those who haven't seen it um regime based Monte Carlo has two sets

19:27

of Capital Market assumptions there's one for the near term and one for the long term

19:32

um you can set you know for example an income lab you can set what you mean by near term we just used a 10-year period

19:39

so near term was 10 years long term was the rest of the plan

How This Worked

19:44

okay so again just to go over how this worked what we did is for each forecasting point

19:52

um we uh we created Capital Market assumptions and we made forecasts

19:59

um for the the period to come um and what we did is we we took 200

20:05

different uh retirement income plans again that were kind of between uh quite

20:10

conservative and quite aggressive so it's sort of you know Tucson versus Tampa um so a a broad range of those and we

20:17

had the model make predictions We Had Each model make predictions about the probability of success of that plan

20:24

right so some were going to be we're going to have very high probability of success one we're going to have very low probability success and lots in between

20:30

and then as we step through time this is you know we just

20:36

um the the each model gets access to more to more of History

20:42

for regime-based Monte Carlo the same except that you know we just extend the

20:47

the amount of History available and same for historical

20:53

right so and we're doing that across you know as much of history as as we possibly could which for us

Results

21:00

um had us making um predictions from 1951 to to 2002

21:08

um and again it was just systematic withdrawal plans from from a 60 40 stock

21:14

Bond portfolio this this kind of study could could very easily be done for other types of plans you know more more

21:21

realistic complex plans as well but this was kind of a a proof of concept and one

21:26

that hopefully we can um extend in the future Okay so

21:33

how did these models perform right that we we have you know thousands of forecasts right for each model

21:39

um what was the the error level well it turned out that regime-based

21:45

Monte Carlo and historical analysis had very similar error levels

21:52

um and both of them were about 25 percent lower than traditional Monte Carlo and the reduced Capital Market

22:00

assumption Monte Carlo so this I mean this is a this is a pretty important finding right so

22:07

um just by kind of improving your forecasting model

22:12

um in doing retirement income planning you can reduce the error of these risk

22:18

estimates um by you know a sizable amount

22:26

so this would be you know kind of like choosing between rain forecasting models

22:31

and just saying hey let's let's do the one it's not perfect right but let's do the one that is that generally has lower

22:37

error and is predicting rain a little bit better rain or dry a little bit better

Calibration

22:44

okay so um this is uh showing us that the

22:51

calibration of these models now I know this is a this is a complicated chart and actually I realize it doesn't it

22:57

doesn't match the the rain forecasting one we had before but what you have here is if if we're above the black line so

23:04

the black line is is um perfect calibration right so every time I predict 80 chance of success I 80

23:11

of those cases succeed um again we don't expect that that to

23:17

happen um and then if you're above the line you're actually getting more success than you

23:23

predicted so this isn't so much what we're looking at here isn't predicting rain it's predicting dry right

23:29

um so if you're above the line um we would have what you would call a

23:35

wet bias so um you're predicting a lower probability of success than you're actually getting

23:42

and just like in Rain forecasting um most people prefer a wet bias there's

23:48

even evidence that that rain forecasts actually have a wet bias whether that's because of the model or because you know

23:54

the the meteorologist um is is doing that on purpose I I don't know but it could even be the latter

24:00

because people would prefer to bring an umbrella and not use it than to not bring an umbrella and wish they had

24:06

um so the wet bias probably is um is something that forecasters prefer

24:11

and that that's likely true for retirement planners as well right so again under promising over delivering is

24:18

what we see above the line um so there's some really notable things

24:24

here if we just look at let's say you know kind of above about 55 probability

24:30

of success um which are the that's kind of the range that most uh plans are going to be in in

24:36

fact most advisors say they actually prefer kind of I think in the 70 to 90

24:42

range is what um Derek I think found in a survey um so he can correct me

24:48

um what we see here is that in particular traditional Monte Carlo

24:55

um which is kind of the most widely used model has a consistent dry bias so it's

25:01

actually um predicting higher probability of success than we actually saw historically so

25:09

it's underestimating risk another way to look at it is using this model would lead to

25:19

um advice for for higher income levels than than a client probably should be

25:26

taking at that at that risk level you know if they have a certain risk appetite

25:32

um regime-based Monte Carlo has a consistent wet bias so it's generally overestimating risk

25:38

um the differences may be uh you know slightly less so it's at a lower wet bias than than traditional has a dry

25:45

bias but but they both have these these pretty consistent biases in in this range

25:51

um historical um tends to be fairly tight around the

25:57

the you know the perfect calibration line but it likes to bounce back and forth and and it has a little bit more

26:02

of a dry bias in this range than it does a wet bias so again maybe a a not the

26:11

kind of under-promise over deliver type of uh type of model foreign

26:17

looking at how predictions came out um in different periods in time because

26:24

what we're looking at here is across the entire period 1951 to 2002

26:30

um and the reason it stopped at 2002 by the way is um we were looking at predictions

26:36

for at least for 20 years um and so that's the last time we could have made a 20-year prediction and found

26:41

out how it turned out um so here we look we're looking at traditional Monte Carlo and the reduced

26:48

um Capital Market assumption version and what we see is you know neither one

26:55

does does particularly well across this entire period um

27:01

you do see that reduced Monte Carlo does particularly well in certain periods and those are the times that turned out

27:07

um were not great times to retire um basically those were times that you needed to be a little bit more

27:13

conservative and so reduced Monte Carlo um of course it's designed to be more conservative so it's kind of a one hit

27:19

wonder right it does well in periods where it turns that that it turns out or not

27:25

um a great periods to retire and are those periods that that we would prefer people be more conservative traditional

27:31

Monte Carlo is is um it does have a couple of times when it's sort of okay but but mostly it's it's among the worst

27:38

um possible ways to do it so in the in the the 60s

27:43

um were kind of a period where um because of high inflation and low real returns that that wasn't a time

27:50

that that really supported really high systematic withdrawals that's where we get the four percent rule from and so on

Regimebased Monte Carlo

27:59

um regime based Monte Carlo and historical on the other hand they're not always the absolute best

28:05

um although regime based gets pretty darn close to being always the best um but uh they they do seem to perform

28:12

better across different kinds of environments right so they do well when

28:18

that in periods that turn out to be poor they do better comparatively in turn in

28:23

periods that turn out to to be great times to retire for example the the mid mid 80s

28:29

um you know turn out to be historically one of the times that people could have taken the highest systematic withdrawals

28:34

because inflation was coming down interest rates were coming down stocks were doing well and so on so none of the

28:42

the plans did you know their best at this period uh none of the models did

28:47

their best at this period but uh regime based and historical we're we're doing the best um you'll also notice in kind of the the

28:54

tech bubble um and you know in other periods region-based was had the lowest errors

29:01

in these time periods okay um

What does this mean

29:07

so that's a lot of data a lot of graphs and things what what is this what does this mean for us

29:13

um first of all um I like a caveat there are other ways

29:18

to to model the world um in fact there are even ways within an income lab that that are not in this

29:24

this study um so I think the first thing this shows us is not all models are the same

29:30

um and there really are differences that would really matter to clients and to the advice that we give clients and so

29:35

it's worth paying attention to the kind of model you're using to model risk and

29:41

to and to give you estimates and forecasts about the future um for for the four that we looked at uh

29:47

it's it's really important to note that traditional Monte Carlo consistently underestimated risk

29:53

at the levels that are most important to planners so you know I'll ask Derek to

29:58

to comment on that in a bit um and that regime based Monte Carlo and

30:04

historical analysis had less error so at least among these four these These are clearly to be preferred when when doing

30:10

retirement planning um it's worth thinking also about this wet and dry bias so what kind of by

30:17

we're going to have imperfections we're going to have errors so what kinds of Errors would we prefer in in the the

30:24

tools that we use to to um back up our advice

30:29

um I think it's at least reasonable that some would prefer a wet bias and and among the the models we looked at regime

30:37

based Monte Carlo um had the most consistent wet bias that gives you the ability to kind of uh bias you toward

30:45

giving good advice through or sorry positive news through retirement right

30:50

hey things are better than we expected maybe most importantly though here no

30:55

model was even close to error free right I mean the the you saw it in the calibration charts but but even just the

31:02

error numbers right the lowest we saw was was 0.12 right um these are not error-free things these

31:10

are not things where you can look at your predictions and say oh I know exactly what someone should do and I

31:15

will be right um they give you um certainly directional information

31:21

they help you give good advice but one interpretation is of this is

31:28

we can't really do one-time planning the errors will be so large we can't have enough confidence to do that and so

31:35

plans should really have a game plan for adjustment and be monitored regularly because as time goes on

31:42

we'll learn more and more about the world that a client is actually living through right so bring it back to the

31:49

the rain analogy you know we can make a prediction about rain a week from now

31:55

um and as we get closer if we never looked at data again we never updated our models we never said oh this is the

32:01

world we're living in um then we would have probably fairly bad forecasts you know the day of right

32:07

but if I'm looking at the the radar right I see the storms approaching or I see you know there are no storms it's

32:13

completely uh clear sky right I can give people better and better advice as as

32:19

time goes on right I mean looking at the sky is is better than looking at a long range forecast so

32:26

I think we need to be humble about the the kinds of information we actually have

32:31

um for for clients and and the kind of the quality of the of the forecast that

32:37

we have we certainly should use the best uh available forecasts um but we also need a a game plan for

32:44

Change and a game plan for um adjusting our uh our forecasts over time

32:52

um so with that I will um bring on Derek here and Derek I don't know if you want to comment about

32:57

kind of the import of this for for planning and uh maybe keep me honest on what the uh what the range that advisors

33:05

prefer is yeah so the the range we saw was really and we're asking about the

33:10

minimum acceptable in that range we ended up finding was 70 to 95 so that

Is traditional Monte Carlo biased

33:16

was kind of the where advisors would put their minimum um and so yeah I do think when you

33:21

really focus in on that range and we look at the results there and we see which models how they're performing and

33:28

how we might you know if a model is going to air in One Direction which direction might we want that to be

33:35

um I think there's a perception out there that a lot of people feel like well I'm going to use traditional Monte Carlo

33:42

um because I want to model you know realities that might be worse than what

33:49

we've seen historically and so I thought that was really interesting to see actually no that you know the let's see

33:55

when we're back testing this that's not actually what we're seeing and the the results is that's actually kind of flipped that a traditional Monte Carlo

34:02

is getting is biased in the wrong direction um if if that's what you're trying to

34:07

aim aim for and I do think having you know that wet bias so to speak overestimating the risk

34:14

um within that range you know it could there could be something that that makes

34:19

sense some people though I could see saying you know I want to maybe I'll account for that web bias in some other

34:25

ways maybe I just want the truest number I can possibly get and at least from this historical might be the Direction I

34:31

Would lean um but then from there you know maybe somebody still is through various ways

34:37

maybe they don't account for home equity maybe they don't account for other types of things that they see as being

34:42

providing some of that protection um that we might still get or want in a

34:48

forecast but yeah overall I just thought it was really fascinating work and to be

34:54

able to for Justin to be able to use the basically the income lab you know the engine and the power behind it to be

35:01

able to go and do this and provide that Insight uh really cool to actually see

35:07

how that would have performed historically and provide some what I

35:12

think is new insight into uh how we how we think about the different models that we use

35:19

Derek in your own um planning have you attended you know

35:25

since you started using income lab um was there an analysis method that

35:30

you've favored have you have used more than one in certain cases

35:35

I would say there are times when I'll you know if I'm really feeling like oh

35:41

do I need to check multiple things here do I need to look at this from different perspectives um I I've used the term kind of

35:47

triangulate before in terms of look at different results and see okay you know from those where where am I get my

35:53

target but honestly one one thing I really like about the historical is

35:59

getting the more consistent results instead of you know like with Monte Carlo

36:04

um and even regime based Monte Carlo getting a new roll of you know guard rails essentially is the plan gets

36:10

updated so I've probably leaned more towards historical almost more from a practicality and this this result made

36:18

me feel good about that but at the same time if I if I was really concerned I'd say for a lot of my clients I'm running

36:24

plans where I feel like there's multiple layers of um kind of protection built in in the sense that I'm not accounting for

36:31

home equity usually I'm usually putting things on very conservative setting things and assuming the longest

36:37

longevity but if I was pushing it more of an edge case type plan then I do

36:43

think um you know maybe the the fact that the regime-based Monte Carlo is going to change from one

36:49

plan or um one run of the plan to the next that still might be worthwhile to get a

36:56

little bit more of that extra conservativeness just built into it yeah I think that's uh that's an

37:03

interesting point so especially like you said if you have clients who

37:08

um you know can can maybe their resources and their spending are are such that they can kind of afford to at

37:14

least start out retirement in the in the kind of lowest risk highest success range here

37:21

um then like you said I mean the the blue line is uh it obviously does have some error

37:27

but it's it's it's very close and then if you have kind of back pocket hey you know we could always tap this other

37:33

thing and and so on um not to mention that you're actually you're starting fairly low risk and so you've already

37:40

um you've already tilted the scale so that the likelihood that they actually have to reduce from that point is quite low

37:47

and the likelihood you might have good news for them anyway is is quite High um so yeah that's that's interesting

37:53

um one thing I noticed on regime based is you know there are particular periods I I actually found

Is historical biased

38:01

this pretty interesting I mean granted the green line is is often the lowest period but

38:07

um the the times where it really Dives down for so for example in um the uh

38:13

the.com era here it tends to do a pretty

38:19

interestingly good job when things are kind of extreme um so you know in this case famously is

38:25

like the highest Cape ever was right so the so the cape filter here is is that is really allowing regime base to say

38:31

hey we're you know do not assume you're gonna get you know 1980s style returns

38:37

from now on um and clearly it you know it turned out to be to be right in this case so it's

38:42

an interesting way to kind of have different Capital Market assumptions in different periods you

38:48

know without it just being you know pure opinion or something like that um it's interesting that historical does

38:53

pretty well even without that filter actually on income lab you can apply a similar filter but even without the

38:59

filter um historical does quite well um I don't know if you have thoughts on

39:05

why that might be I I mean the my only thought on in terms

39:11

of why historical sometimes does perform well in a lot of situations like these does come back to it's just a more

39:18

compared to Monte Carlo it's a more realistic um kind of Market behavior in terms of

39:24

we see mean reversion we see momentum we see some of those effects that you don't actually you know that I mean you you

39:31

could program them into Monte Carlo simulation like that could be done but generally speaking

39:36

um it's just a random draw from one time period to the next time period And there's no

39:42

correlation within time periods that we're taking a look at and so I think that to me is one reason why maybe we

39:48

see new historical uh performs well is it captures some of that real world type

39:55

of type movement yeah yeah I agree that that could be true like you you know we put in Capital

40:01

Market assumptions you've got even with regime based you just have two sets right near term and long term and for

40:07

each of those you have an average standard deviation and you got a correlation Matrix well in reality what the joke is right the only thing that

40:13

goes up in uh bear markets is correlations right so you're not able to say hey correlations will change too

40:20

over time and so on and so historical does just kind of capture whatever whatever the

40:26

um the the relationships between assets uh can be in the real world

40:34

okay um I think looks like we've got

Questions

40:39

quite a few uh questions here Taylor yeah

40:45

um so some of them uh can be answered together I guess uh the first one is

40:52

um have this has this research been published somewhere uh Derek and I have a draft that's

40:59

that's being edited right now um so I yeah hopefully you'll see it

41:05

someone soon but no this would be a webinar exclusive here you're getting a first look

41:10

haha good answer Derek um the next one is um how do you think

41:16

this research applies to building plans and income Labs specifically using historical versus regime

41:23

Etc does it make you want to build plans using any model specifically

41:28

I might take this one just for my I guess the practitioner perspective I.E the way I look at it

41:34

um I I'd always kind of lean regime based and historical before

41:40

um between those two and as I mentioned historical I I also like for some other reasons just in terms of getting

41:46

consistency between iterations of a plan I might be running but um I think that

41:51

to me only kind of solidifies that that now I lean even harder on using either

41:58

regime based or historical as my preferred method and again there's

42:03

different contexts or situations I might use one or the other but those would be the two I would tend to go to

42:11

yeah and I would I guess I have some some thoughts on that as well I think I always

42:16

um had a bias toward historical because of you know what we were just talking about with how you're kind of capturing

42:22

changes in correlation and really modeling a really wide range of possible worlds and worlds that we know can exist

42:29

because they have existed um this increased my confidence in regime

42:35

based Monte Carlo um I I personally like the wet bias I think that it's nice to be able to

42:41

um kind of uh give people um good news um and and I don't think that the wet

42:47

bias is so extreme that you're um you know kind of giving way out wrong answers or anything like that I've also

42:54

seen in practice um that there are certain plans that could really benefit from regime based

43:02

over historical the the main example I can think of there is um plans that are really subject to

43:09

inflation risk so if you're using regime based and you're able to say no look in

43:14

the first 10 years or five years or whatever your near-term period is let's assume inflation's really high then

43:19

you're able to kind of say hey let's let's do a plan that for this particular risk I I have kind of been conservative

43:27

if you're using historical and you're not filtering it at all which which again an income might be actually can

43:32

filter but um then you're getting periods of time with low inflation periods of time with high

43:38

inflation periods of time with deflation so you're getting a really cool broad range of things but you're including in

43:44

the simulation some periods that um You probably don't think are likely to happen now we could be wrong right

43:50

but um you know that that's one place I've really seen regime base be regime based uh be be useful for for getting

43:58

um getting at the issue that somebody has so a good example of that is if if you have a lot of pension income that's

44:05

not adjusted for inflation um that could be a plan that's really subject to inflation risk and so you

44:11

know you might want a particular tool to to help give those clients really good

44:16

um targeted advice and just to attack quickly onto that too as well I think uh you know the economic

44:22

environment we're in can be another reason where like a lot of people were concerned to go back a year maybe not

44:28

not so much now but a year ago with high Mark evaluations low interest rates uh

44:33

you know the wanting to capture some of that in their Capital Market assumptions so when there's particular economic

44:40

concerns uh just similar to Justin's point there with certain plan types I think that could be another reason where

44:46

regime based Monte Carlo is is a much better approach than this across the

44:52

board reduce everything Monte Carlo which clearly just gets you know too far

44:57

off of reality in my opinion yeah I guess we didn't address that one

45:03

too much I mean definitely it it moved the the dry bias it got rid of most of the dry bias but

45:09

um really it like I said it's kind of a one-hit wonder so it only works when when things are going to be bad right

45:15

because you're so it doesn't work kind of it's not an all-weather type model

45:21

um all right so um just a reminder that we do have some

45:27

questions in here if you'd like to hear the answer you can vote for them uh we do you know we have are limited on time

45:33

so I wanted to make sure that the most important or most uh voted on questions are getting answered uh there is one

45:41

that says when um when referring to I guess we can go back to to Derek's

45:46

question it says Derek do you discuss the pros and cons of different models with clients or do you triangulate

45:52

behind the scenes almost always behind the scenes if a particular client wants to dive into it

45:58

I want to be prepared and you know ready to explain that but yeah it is very rarely that I'm getting into any of the

46:06

type of stuff we're talking about here today with a client I'm usually focused much more on the guard rails

46:11

um and those levels so that they just understand here's when an adjustment will be called for

46:18

okay uh the next one is have you extended the rolling five-year analysis to and Beyond 2007 to 2009 and I think

46:27

that's kind of partnered with speculate how 30-year analysis might have scored out

46:33

Yeah we actually because we were doing 20-year forecasts in order to to get at

46:38

least to the tech bubble we also did the exact same study with 30-year forecasts and for the period that overlaps right

46:45

so this that only let us get through 1992. um

46:51

the results for 20-year and 30-year forecasts were very very close so that

46:57

like the the correlation between those results was was very close which you know we we won't know uh how 30-year

47:04

forecasts in 2002 really work out until 10 years from now but it does give me confidence that given that they were

47:11

very similar for the periods that do overlap um I I feel relatively comfortable

47:16

saying hey these models might perform similarly for 30-year forecasts we didn't do it for 10-year or five year

47:22

forecast that that would obviously be a great way to extend This research um extending it to other models right I

47:30

mentioned being able to filter history or you know you might have other ways of creating Capital Market assumptions

47:36

um other types of plans is an obvious one right plans with Social Security

47:41

plans with the retirement Hatchet that Derek and I have talked about before plans with um the retirement smile all

47:48

of those kinds of things are you know relevant for for planners because you're dealing with real situations that that

47:54

really do have those characteristics so I think that's a direction we'll go at some point but in presenting research

48:00

you know it's usually a good idea just to use a very simple um simple uh approach

48:07

so when it comes to the Monte Carlo models and the question is are what are

48:13

other you know tech companies using Wireless e-money riskilize others and

Income Lab Results

48:18

are these results actually income lab results foreign

48:27

these are in a sense actual income lab results because these are models that

48:33

are available um inside of income lab so you can do historical analysis you can do regime

48:39

based Monte Carlo you can do traditional Monte Carlo and you could go in and customize your traditional Monte Carlo

48:45

and take take two percent off um so in that sense they are they are

48:51

results that you know is basically saying if you had used income lab to you know make forecasts at each

48:58

monthly point in time between 1951 and 2002 um you know these would have been the

49:04

outcomes there is something missing from this entire discussion which is um you know income lab doesn't depend on

49:11

success and failure it depends on adjustment so so people wouldn't have just you know set a plan going and done

49:17

nothing um they would have adjusted along the way so um in that sense you know plans wouldn't

49:24

have succeeded or failed they would have adjusted so I think this is this is maybe a little too abstract to

49:30

say that their income lab results we actually do have some other um things coming out next month uh in

49:37

beta that that deal with that question of like what would actual income lab results with adjustments and so on I've

49:43

looked like so uh you'll have to come to our next webinar for that one

49:50

um Derek I don't know on the other Tech um side of things I mean we certainly tried to

49:56

use um to test models that are actually you

50:02

know close stand-ins for things that are being used in the industry right we wanted this to be practical um but I don't know that we have you

50:08

know um a list of who uses what in the yeah I mean we didn't necessarily map it to

50:15

like here's every providers what they do in terms of how they do Monte Carlo but I think just in general

50:22

um you know the most platforms it's just a traditional Monte Carlo uh probably

50:27

where the variation is is what Capital Market assumptions do they default to and how is that set that can even vary

50:34

by firm or company level if your firm or company has a particular set of Capital

50:40

Market assumptions they want to use so a lot of variation um there but we you know again tried to

50:46

try to get the traditional Monte Carlo here to just be as kind of neutral

50:52

common type of a application of traditional Monte Carlo

50:58

so partnering with that when referring to utilizing historical Monte Carlo the inputs for traditional software money

Historical Monte Carlo

51:05

guide Pro right Capital would be to default the historical asset class returns within the program rather than

51:11

projected returns yeah I think our our

51:16

two kinds of traditional Monte Carlo that you know what we call traditional and reduced Capital Market assumptions

51:22

we're meant to be stand-ins for those two approaches so typically you will get

51:28

default assumptions that are based on history up to that point 30-year averages are pretty common

51:35

um but there's nothing magic about those there might be a firm that does you know 40 or 50 year averages

51:41

um and then the reducing it by two percent was meant to be a stand-in for

51:46

um forecasts um so we don't expect that typically

51:51

forecasts uh are to be you know two percent higher than history

51:57

um at least in recent history what I've what I've generally seen is that they're more muted um and so that was kind of our stand-in

52:04

for for you know using kind of your favorite Capital Market forecasts

52:10

and one one just kind of side note on that but I think is relevant though and Justin correct me correct me if I'm

52:16

wrong but I believe that like when you look at some of the forward-looking assumptions one that's pretty popular is like the JP Morgan

52:23

um Capital Market assumptions but those are 10-year assumptions correct Justin

52:28

those are they are um you know I haven't looked at them recently and that is yeah one of the

52:36

one of the reasons I think people have have liked at least researchers have liked the idea of regime-based Monte

52:41

Carlo although I as far as I know income lab is the only place that's actually available um is is exactly that it's not hard to

52:48

find forecasts but they don't tend to be 30-year forecasts or you know 40-year forecasts or however long your your plan

52:54

might be 20-year forecasts they tend to be kind of medium term forecasts

52:59

um and so you know maybe you have a forecast for lower um I don't know Bond returns

53:05

um and you might even be right but it's not clear you want that to be your forecast for 30 years you might just

53:11

want it to be your forecast for five eight ten years and then revert to something else afterwards so

53:18

um that's uh one reason I think researchers

53:23

have liked um regime based Monte Carlo and and um I think the evidence from

53:29

that we presented here is that actually also in practice it might be a a viable option and definitely one that seems

53:35

better than um just straightforward traditional Monte Carlo yep and I just mentioned

53:42

that because I know that's one that commonly sometimes advisors I think may not realize oh wait this is a 10-year

53:48

forecast it's not a 30 year so whatever Capital Market assumptions you're using um you do want to be careful to make

53:54

sure you understand what they're what the time frame is intended for those as well

54:01

I saw a question about where you do this and um income lab so I just thought I'd show

54:06

everybody um if you want to change in a particular plan what kind of analysis you're doing

54:14

go to the plans advanced settings go to plan analysis and you can change it right here

54:21

you can also change the default that you want for your for all new households that you create

54:29

by going to settings default values the default in this you know just

54:35

fake account here is historical but you know if you want to always use regime based you can just save that that won't

54:42

change any existing plans or households so this would only be for for new ones

54:51

but you can change it plan by plan and change it and my plan by going to the plant's advanced settings yeah

54:57

and if you copy a plan it's it's copying that setting from the plan you're copying so it's

55:04

um you're not getting that new default setting and there's no way to combine them there's no combination

55:12

okay uh we have talked probably for one or two more questions this is in

Regime Based Analysis

55:18

follow-up to that when using regime based analysis and income lab does the user have to provide the 10-year

55:25

projections or can income lab provide or suggest them based on similar periods in history

55:31

yeah so that's what we do um actually there's an article about this in our knowledge base but

55:36

um the defaults and income lab are are created formulaically so it's not like

55:43

um you know Derek and Justin sit around and think huh what do we think you know will happen over the next 10 years it's

55:49

pure formula so we take all of history so it's it's basically this um

55:57

uh we take all of history and then we filter it

56:02

um for economic context now we don't just use Cape we we use kind of a conglomerate of economic factors and we

56:10

say hey you know the early 80s were not like today because Cape was really low and

56:17

you know interest rates were 14 and things like that and then using those periods of History

56:23

we um we create the near term and the long term

56:29

um assumptions near term being returns that were close you know in time to those times that were like today and

56:37

um long term being the next periods so and we update those every month um and again it's totally formulaic

56:46

um you can do custom um and create your own it's just a toggle

56:51

um and then if you go back to default you'll you'll reload the um the the defaults that were created

56:57

um from historical averages all right well I

57:05

I think we have time for maybe one more said what would have to change in the real world for regime based to be

57:10

questionable going forward I know you mentioned inflation Justin but anything further on that

57:18

foreign think about ways that regime based would

57:25

be was the question for to be questionable going forward so I think what you would

57:30

have to have for for anything where there's like a filter being used which is how we do how we create regime based

57:36

uh assumptions um look if your regime based assumptions are turned out to be good there's really

57:43

nothing to make regime based bad right I mean the assumption is that your your Capital Market assumptions were good so

57:48

the way that regime based wouldn't work is if the Capital Market assumptions turn out to be just just bad just wrong

57:54

so for example if we're filtering with cape we're filtering out low Cape periods well locate periods were

58:01

generally followed by good stock returns um so we're filtering out the periods that had good stock returns which means

58:07

your near-term assumptions are generally going to be lower stock returns if it turns out that really high stock returns

58:14

are are you know coming even though Cape is already really high um that those would be bad assumptions

58:19

right we would get we would get bad performance so basically things would have to kind of flip on their heads from

58:25

how they've been in in history um for for regime based to perform poorly and you know could that happen uh

58:33

possibly um but it uh you know history is the only data that we actually have so um it's uh

58:40

it's uh I would probably be uh reluctant to it to assume that will definitely

58:46

happen yeah I would just add to that you would only know that after the fact and I still would probably lean towards using

58:52

some of that Insight the economic context to plan based on what we know in the moment

58:58

um and then the other other way of thinking about that question I guess a little bit different but if we're in

59:04

just a like perfectly average time where the economic context isn't actually doing anything you you might be in a

59:11

situation where you're getting basically the same result it's just not doing as much but um it wouldn't hurt

59:17

yeah yeah and to that that point also I think this is where um

59:22

having a game plan for for adjustments like this is okay this is what we're gonna do and we'll do this until we have

59:29

clear reason to change um and then having a plan for what those changes would be when they'll happen how

59:34

big they are and so on um that kind of makes getting this number exactly right

59:41

um you know a little less important and that's that's good because these numbers

59:46

are not exactly right uh that just can't be that again that was sort of the

59:52

um probably the the second biggest takeaway from from this work is just that

59:58

um there's there's always error and so um if the future turns out worse than

1:00:04

the past um we'll learn that as we go on in time and we'll adjust to it to accommodate

1:00:09

that we can't hope to predict it ahead of time all right well we are at the top of the

1:00:15

hour so I figured we would wrap this up um this has been recorded and it will be available and distributed uh via email

1:00:23

to the attendees and the folks that registered uh there was just a follow-up if this when you publish This research

1:00:29

do you know where it would be published uh just uh thought that that you might

1:00:34

be able to answer that Derek or Justin people people might know where we've

1:00:40

published research in the past commonly I I don't know I don't want to say too much before it's gone through any processes or evaluated but

1:00:49

um common places where you see me write stuff might be a good place to go check out

1:00:54

we'll announce it from The Mountaintop

1:01:00

for attending again we do have a lab talk Tuesday the third Tuesday of every month and then an educational webinar

1:01:06

the fourth Tuesday of every month uh thank you all for being engaged with your questions and if you need anything

1:01:12

further uh contact us at info incominglaboratory.com

1:01:20

thank you