X - MUCM

Perfunctory sensitivity

analysis

Andrea Saltelli

European Commission, Joint

Research Centre, Ispra (I)

UCM 2010 Sheffield

andrea.saltelli@jrc.ec.europa.eu

Has the ‘crunch’

something to do with

mathematical modeling?

August 2009 issue

Here is what killed your 401(k)…

Li’s Gaussian copula function …

Felix Salmon, Wired, February 2009

Useless Arithmetic: Why

Environmental Scientists Can't

Predict the Future

by Orrin H. Pilkey and Linda

Pilkey-Jarvis

‘Quantitative mathematical

models used by policy makers

and government administrators

to form environmental policies

are seriously flawed’

We just can’t predict, concludes N. N. Taleb, and we

are victims of the ludic fallacy, of delusion of

uncertainty, and so on. Modelling is just another

attempt to ‘Platonify’ reality…

Written before the

financial crisis

Nassim Nichola

Taleb, The Black

Swan, Penguin,

London 2007

Do we need better models?

How about better ways of using them?

.. where ‘better’

sides …

has both normative and technical

‘Sensitivity analysis would Help’

Edward E. Leamer, UCLA

Edward E. Leamer, 1990, Let's

Take the Con Out of Econometrics,

American Economics Review, 73

(March 1983), 31-43.

E. E. Leamer, UCLA

Peter Kennedy’s ‘A Guide to

Econometrics’.

Anticipating criticism by applying

sensitivity analysis. This is one of the

ten commandments of applied

econometrics:

“… my observation of economists at work who

routinely pass their data through the filters of

many models and then choose a few results for

reporting purposes.“

Tantalus on the Road to Asymptopia

Edward E. Leamer, 2010 Journal of Economic Perspectives, 24, (2), 31–46.

Party’s skunk

syndrome

“One reason these methods are rarely used is

their honesty seems destructive.”

Ibidem

My preferred methods: Variance

based

( ( ) ) E Y X

VX i X~

i

First order effect, or top marginal

variance=

= the expected reduction in variance

than would be achieved if factor Xi

could be fixed.

i

EX ~ i i

Total effect, or bottom marginal

variance=

= the expected variance than would

be left if all factors but Xi could be

fixed.

( ( ) ) X

X Y V

~ i

From this … … to this

( ( ) ) E Y X

VX i X~

i

This is a first

order effect, or

top marginal

variance.

i

EX ~ i i

This is a total

order effect, or

bottom marginal

variance.

( ( ) ) X

X Y V

~ i

( ( ) ) E Y X

VX X~

i

i =

V ( Y )

EX ~ X i

( ( ) ) V Y X

~ i

i =

V ( Y )

i

Rescaled to [0,1], under the name of first order and

S

i

S

total order sensitivity coefficient

Ti

Why these two measures?

( ( ) ) E Y X

VX i X~

i

EX ~ i i

( ( ) ) X

X Y V

i

~ i

Factors prioritization

Fixing (dropping) non

important factors

Even when factors are not independent!

See: Saltelli A. Tarantola S., 2002, On the relative importance of

input factors in mathematical models: safety assessment for

nuclear waste disposal, Journal of American Statistical

Association, 97 (459), 02-709.

Latest works

(2010)

Critical use of SA

From: Saltelli, A., D'Hombres, 2010, Sensitivity

analysis didn't help. A practitioner's critique of the

Stern review, GLOBAL ENVIRONMENTAL CHANGE,

20, 298-302.

The case of Stern’s

Review – Technical Annex

to postscript

William Nordhaus,

University of Yale

Nicholas Stern, London

School of Economics

Stern, N., Stern Review on the Economics of Climate Change. UK

Government Economic Service, London, www.sternreview.org.uk.

Nordhaus W., Critical Assumptions in the Stern Review on Climate

Change, SCIENCE, 317, 201-202, (2007).

Stern’s Review – Technical Annex to postscript (a

sensitivity analysis of a cost benefit analysis)

The Stern -

Nordhaus

exchange on SCIENCE

Nordhaus falsifies Stern based on ‘wrong’

range of discount rate (~ you GIGOing)

Stern ‘My analysis shows robustness’

My problems with it:

… but foremost Stern says:

changing assumptions important effect

when instead he should admit that:

changing assumptions all changes a lot

How was it done? A reverse engineering of the analysis

% loss in GDP per capita

Same criticism applies to Nordhaus – both authors

frame the debate around numbers which are …

… precisely wrong

Are statistical

practices for SA

taken up?

From: Saltelli, A., Annoni P., 2010

How to avoid a perfunctory sensitivity analysis,

Environmental Modeling and Software,

doi:10.1016/j.envsoft.2010.04.012.

Who do these have in common?

J. Campbell, et al., Science 322, 1085 (2008).

R. Bailis, M. Ezzati, D. Kammen, Science 308, 98 (2005).

E. Stites, P. Trampont, Z. Ma, K. Ravichandran, Science

318, 463 (2007).

J. Murphy, et al., Nature 430, 768-772 (2004).

J. Coggan, et al., Science 309, 446 (2005).

OAT

OAT in 2 dimensions

Area circle / area

square =?

~ 3/4

OAT in 3 dimensions

Volume sphere /

volume cube =?

~ 1/2

OAT in 10 dimensions

Volume hypersphere / volume

ten dimensional hypercube =? ~ 0.0025

OAT in k dimensions

K=2

K=3

K=10

In 3 dimensions, OAT,

7 points

This is what is done

In 3 dimension, 8 screening

points in a trajectory

arrangement

This is what could be done

This is a screening good practice

(Method of the Elementary

Effects)

See: Campolongo, F., Cariboni, J., and Saltelli, A., 2007, An

effective screening design for sensitivity analysis of large

models, Environmental Modelling and Software, 22,1509-

1518.

One could also do using OAT’s

instead of trajectories.

See: Campolongo F, Saltelli A, Cariboni, J, 2010, From

screening to quantitative sensitivity analysis. A unified

approach, Submitted to Computer Physics Communication.

Increasing the

number of trajectories

from tens to hundreds

the test becomes

quantitative.

It is the same

the design

used for the

total sensitivity

index.

See: Saltelli, A., Annoni, P., Azzini, I., Campolongo, F.,

Ratto, M., Tarantola, S., 2010, Variance based sensitivity

analysis of model output. Design and estimator for the total

sensitivity index, Computer Physics Communications, 181,

259-270.

Thus one can start screening wise (few points)

and continue variance-based, without discarding

points, by just changing the estimator.

SA Conference and Course

Sixth International Conference on Sensitivity

Analysis of Model Output, Bocconi University

of Milan, 19-22 July 2010.

Sixth Summer School on Sensitivity Analysis

of Model Output, Villa La Stella, Fiesole -

Florence, 14-17 September 2010.