Decisions under Uncertainty. Part 1

TLDR

For many of you reading this, developing and testing predictive models will be a lot of fun - but my pragmatic, buzz-killing message here, is that we shouldn’t forget why we’re asked to do it. I suggest that the reason is (almost?) always, to help support decision-making. And yet, we often have this unhelpful disconnect between a calculation, and what the results imply for how we should act. An analyst hands over their work to a manager or budget holder, who then reinterprets everything and makes a call, which may or may not be consistent with the information they have been provided - sound familiar? It does to me.

In this post (Part 1) these two domains are considered jointly, using an example. We see how this can allow for practioners to arrive at coherent (quantitiative) and reproducible (auditable) decisions. Part 2 will consider a specific application of this approach, known as value of information analysis.

As ever, all code (in Julia today) is freely available!

---

A Familiar Scenario

When I worked as an engineer I would often be asked to perform calculations (using spreadsheets 🙃 ) to arrive at some number regarding a safe operating condition for a damaged structure.

Models for the sake of themselves can be plenty of fun, but unless they are tied to the underlying decsions that they are supposed to inform, they are not necessarily fulfilling their purpose.

So You’ve Fit a Predictive Model?

How might we bridge this gap? How

flowchart LR
  d1[Decision] --> o1((Uncertain \noutcome))
  d1 --> c1{Utility}
  o1 --> c1

Example influence diagram with decisions (square nodes), uncertain outcomes (circles) and utility (diamonds).

Analysis > Vibes!

These (decision/influence/whatever!) diagrams provide a clear map of what factors we think we need to consider when making a decision. They also provide causal information, showing how intervening at one point can have downstream effects. They may seem trivial, but once you try and draw one for a problem you are working on, you will see they encourage you to think deeply about your area of expertise - no seriously, please try it!

Their apparent simplicity also has the benefit of being able to communicate qualitative information with various stakeholders. An experienced engineer could map out complex relationships between variables so that a statistical model could be developed that was consistent with this information. This could then be used to inform decisions.

But, is any of this ever required? If the goal is to arrive at a decision that aligns with subject matter knowledge, then can’t we just ask the expert to jump right to the decision?

This will not always be a valid shortcut. If you have attended workshops that were organised to get a room of experts to agree on a strategy, there will not always be a clear agreement. This is not necessarily because of a lack of expertise, but because of a lack of a common language to communicate the information. Formalising the structure of the problem, the prior knowledge, and where data should be used to update parameters results in a quantitive, reproducible and (with a little extra work) explanaible decision-making process.

Promising vs. Delivering: An Example

Intro

Lets consider a simple example…

An urn is filled with 10 balls, 5 red and 5 blue. As part of a lottery game, you are invited to draw a ball…nah, I’m just kidding. We’re not doing that. Let’s all please stop doing that.

You’re starting a business and have a prototype handmade product that is gaining some interest. You’ve pitched to some retailers and you have a couple of contracts on the table. However, these big shops are covering their backs with a penalty clause - if you deliver the order on time, you’ll be paid, if you over-promise (under-deliver) you pay a penalty to cover their ‘downstream’ losses.

Some data

To help make this decision, lets draw it out. Below is a diagram where we have listed all of the factors that we would like to consider, and how they are related.

flowchart LR
  d1[Decision] --> o1((Uncertain \noutcome))
  d1 --> c1{Utility}
  o1 --> c1

Small business decision problem: which contract should you accept?

…or not.

We may not necessarily have data on the rate at which you can produce your product. Focussing on decisions, rather than models reminds us of this. Sometimes a decision is required before we complete the experiments, trials, surveys etc. that we would like to.

Maybe you’ve not been feeling confident in yourself lately, and are too worried to take on any risk. Perhaps some well meaning friends have tried to counter this by telling you that you can achieve anything. This may convince you to take whatever orders are available. It’s easy to be swayed either way by such circumstances, and anecdotes and intuition drive decisions at much larger companies too.

# for working with data
using CSV, DataFrames, DataFramesMeta

# for fitting probabilistic models
using Random, Distributions, Turing

# for optimisation and solving decision problems

using JuMP, HiGHS, DecisionProgramming

[ Info: Precompiling LogDensityProblemsADForwardDiffBenchmarkToolsExt [501e9729-9453-5dfd-bf52-aa40e3634be9]

MersenneTwister(231123) |>
  prng -> rand(prng, Normal(0, 1), 10)

10-element Vector{Float64}:
 -0.4797047482429591
  0.3492046160452603
  1.9035942889710349
 -1.0599399228488406
 -0.05026230935484321
  1.6159635845804887
  0.46192798081799236
  0.38816396593759556
 -1.4500821161955104
 -0.3667444519178413

Reproducibility

Might your decisions impact the safety of the public?