recap: Enzyme is great 🧪

As we saw in Part 1, Enzyme solves the reliability problems that Zygote can (occasionally) exhibit. It differentiates our code at the LLVM level and is mega performant.

We also discussed in Part 1, how nice it is that Julia projects use only Julia code and the benefits of this for interoperability. This isn’t strictly true for Enzyme. Since it operates at the LLVM level, it can differentiate code in any language that compiles to the LLVM IR (including C++, and Rust!). We call a Julia API, but the differentiation is actually happening in the Enzyme software, outside of Julia.

So how good can autodiff be, if we learn lessons from Zygote and Enzyme, but stay entirely in Julia….

enter Mooncake 🥮

The pitch is as follows: an AD library, written entirely in Julia and competitive with Enzyme.

Like Enzyme, Mooncake handles mutation, control flow, and provides reliable correctness - stuff that Zygote can struggle with.

…but unlike Enzyme, it does all of this without leaving Julia. It is a self-described language-level autograd compiler.

Why does staying in Julia matter?

• debugging: we get Julia errors - not messages from LLVM-land, which I certainly can’t follow.
• new custom rules: adding new derivatives just requires a Julia function. Mooncake provides helpful macros for this too!
• stability: we won’t get breaking changes on new Julia releases if something changes with the LLVM. I have read that Enzyme has previously had to make fixes for this.

If you still aren’t sure whether to take notice, then listen to the man himself, Chris Rackauckas, “Mooncake is Zygote, but done good, with mutation support” (see clip below)

lets be honest about current trade-offs

It’s tricky trying to find definitive benchmarks for the Julia AD ecosystem. The discourse pages will provide one-off examples of older libraries occasionally outperforming newer ones in terms of speed and reliability. However, the general consensus seems to be that Enzyme is currently the most performant, with Mooncake not far behind.

Let’s do our own….

# loading our autodiff libraries
using ReverseDiff, Zygote, Enzyme, Mooncake

# and to test
using BenchmarkTools, Random

# a simple function
test_function(x) = sum(sin.(x) .+ cos.(x.^2))

# and an example with some control flow (the *if*'s can be a problem for some AD libraries)
function control_flow_function(x)
    s = 0.0
    for i in eachindex(x)
        if x[i] > 0.0
            s += sin(x[i])
        else
            s += cos(x[i])
        end
    end
    return s
end

# simulate some reproducible inputs
x = rand(MersenneTwister(2311), 1000)

@btime ReverseDiff.gradient(test_function, $x)[begin]
@btime ReverseDiff.gradient(control_flow_function, $x)[begin]

@btime Zygote.gradient(test_function, $x)
@btime Zygote.gradient(control_flow_function, $x)

@btime Enzyme.gradient(Reverse, test_function, $x)
@btime Enzyme.gradient(Reverse, control_flow_function, $x)

Unlike Enzyme, which has both gradient() (convenience) and autodiff() (more explicit specification), Mooncake’s value_and_gradient!! is the main API. There isn’t a separate “advanced” form.

# Mooncake needs a prepared cache
cache = Mooncake.prepare_gradient_cache(test_function, x);
@btime Mooncake.value_and_gradient!!(cache, test_function, $x)

cache = Mooncake.prepare_gradient_cache(control_flow_function, x);
@btime Mooncake.value_and_gradient!!(cache, control_flow_function, $x)

The results… 🥇 🥈 🥉

test_function()

control_flow_function()

As expected, Enzyme is fast. It sees this already-optimised LLVM code and differentiates that.

Although Mooncake was the slowest on the simple test function, it was only times slower than Enzyme - same order of magnitude, still competitive. The newer libraries really shone on the control flow function, hundreds (Mooncake) to thousands (Enzyme) times faster than Zygote.

And look at the memory! 👀

Mooncake allocates so little! After the initial cache preparation, the gradients calculated in training or inference loops will be almost zero-allocation, which is fantastic for performance or memory bottlenecks.

reviewing the Bayesian model from Part 1

Using the same simulated data and priors, let’s run the linear regression using Turing (look how much neater it is) and swap AD backends with a single argument.

using Turing, Distributions

@model function linear_regression(x, y)
    # priors
    α ~ α_prior; β ~ β_prior; σ ~ σ_prior
    
    # there are ofc lots of ways to vector/optim-ise the likelihood, but...
    for i in eachindex(y)
        y[i] ~ Normal(α + β * x[i], σ)
    end
end

linear_model = linear_regression(x, y); n_draws = 1_000

just running a single chain for post-warmup samples for the purposes of this example:

mooncake_draws =  sample(linear_model, NUTS(; adtype=AutoMooncake(; config=nothing)), n_draws)

enzyme_draws = sample(linear_model, NUTS(; adtype=AutoEnzyme(; mode=Enzyme.set_runtime_activity(Enzyme.Reverse))), n_draws)

not ideal syntax, but within each sample you can specify an AD backend. Actually in the above example, I wouldn’t expect any benefit from using Mooncake or Enzyme - it’s a tiny model ( parameters, data points) and, as discussed in Part 1, a Forward Mode AD library like ForwardDiff would likely be the best bet. As I mentioned in Part 1, there have been cases where I benefitted enormously from simply switching the AD backend to AutoMooncake().

my current thoughts

Please keep in mind, I don’t feel best placed to comment on direction of travel. I am a user (and fan) of the Julia scientific computing ecosystem, but I am not an open source developer.

Mooncake is still being developed by that community, with the stated goal to: “improve on ForwardDiff.jl, ReverseDiff.jl, and Zygote.jl in several ways.” It encourages us to use it, in a seemingly arduous way. Either by adding an extra step (prepare_gradient()), or by using DifferentiationInterface - a common interface for multiple Julia AD libraries.

As with most statistical software, I expect the future success of these libraries will be tied to how well they integrate with the rest of the ecosystem. If Mooncake can be subbed in for Enzyme in libraries like Turing and Flux, then it will be easy to switch and benefit from its features.

Or will the Julia AD ecosystem fail to converge and I’ll end up writing a Part 3 in this series? 🤔

Day 1: a password from a decoy safe 🔓

So we have this safe, with a dial set to 50, and a sequence of turns to make, see below:

seq = ["L68", "L30", "R48", "L5", "R60", "L55", "L1", "L99", "R14", "L82"]

Here an “R” means we are turning the dial to the right, and an “L” means we are turning it to the left, followed by the number indicating how many turns. “L68” means turn the dial left 68 times. The dial wraps around, so if we turn left from 0, we go to 99, 98, and so on.

We are asked:

• part 1: how many times do we end up on zero after a turn?
• part 2: how many times do we turn through zero in total?

Floor division fld() will count how many times we lap around zero (and in which direction).

function get_passwords(seq::Vector{String}, dial_start::Int = 50, dial_mod::Int = 100)

    dial = dial_start; pass_zeros = 0; end_zeros = 0

    for turn ∈ seq
        direction = turn[begin]; turns = parse(Int, turn[2:end])
        
        if direction == 'R'

            pass_zeros += fld(dial + turns, dial_mod)
            dial += turns

        elseif direction == 'L'

            # we need to catch the case where the dial starts at 0 and not count that first turn
            pass_zeros -= fld(dial - turns - 1, dial_mod) - fld(dial - 1, dial_mod)
            dial -= turns
            
        end

        dial = mod(dial, dial_mod)

        if dial == 0
            end_zeros += 1
        end
    end

    return (end_zeros=end_zeros, pass_zeros=pass_zeros)
end

get_passwords (generic function with 3 methods)

A nice opener, and I did make a mistake in my first guess for part 2, forgetting the case where the dial starts on zero and we turn left. We shouldn’t count that first turn as going through zero, and so I had to add the start and end points, with that -1 offset.

It appears I wasn’t the only one…

[2025 Day 1 Part 2] Public Service Announcement
byu/StaticMoose inadventofcode

seq |> get_passwords

(end_zeros = 3, pass_zeros = 6)

⭐⭐

Day 2: fake IDs at the gift shop 🎁

We’re given these ‘product IDs’ from the gift shop, and we need to sort through them to find which ones are invalid. Problematic IDs that we are looking for are made up exclusively of repeating sequences of numbers.

product_ids = ["11-22", "95-115", "998-1012", "1188511880-1188511890", "222220-222224",
        "1698522-1698528","446443-446449","38593856-38593862","565653-565659",
        "824824821-824824827","2121212118-2121212124"]

In part 1, we were looking for a single repeated sequence, so the first half of the ID is the same as the second half. Then in part 2, the length of that sequence could change. Here’s my solution:

function is_invalid(id::Int; part_1::Bool = false)
    
    id_string = string(id); midpoint = length(id_string) ÷ 2

    if part_1
        return id_string[begin:midpoint] == id_string[midpoint+1:end]
    end

    for pattern_length ∈ 1:midpoint 
    # a pattern more than half the length can't repeat
        if id_string == id_string[begin:pattern_length] ^ (length(id_string) ÷ pattern_length)
        # is the id made *only* of this repeated pattern?
            return true
        end
    end

    return false
end

is_invalid (generic function with 1 method)

The puzzle itself felt simpler than yesterday, but only after I read the description a few times. I initially thought that we were going to be asking to find IDs that contained some (any) repeated sequence, but knowing that the entire ID had to be a repetition simplified things.

invalid_ids = Int[]
for id ∈ product_ids

    parts = split(id, "-")
    id_start = parse(Int, parts[begin]); id_end = parse(Int, parts[end])

    for id ∈ id_start:id_end
        if is_invalid(id)
            push!(invalid_ids, id)
        end
    end

end 

invalid_ids |> sum

4174379265

BenchmarkTools.jl tells me this runs in 459.768 ms, with lots of allocations 606.35 MiB, presumably due to all of the string splitting, concatenation, and going back and forth to integers. I don’t think I’ll have time to think about smarter ways to do this, but I’m sure I’ll be forced to in the coming days. Happy to be brute-forcing for now.

My young children happened to wake me up super early today, so I will have scored highly on the leaderboard, but I’m in it for the puzzles, not the glory.

⭐⭐

Day 3: maximum joltage ⚡

We find batteries, each with their own joltage rating, arranged in banks (rows).

Going from left to right, which batteries should we connect up to get the maximum joltage possible from each row?

joltages = [
    "987654321111111"
    "811111111111119"
    "234234234234278"
    "818181911112111"
    ]

For instance, if we were asked to connect 4 batteries in the third row, we can’t make a 4 digit number higher than 4478 (selecting batteries 3 (joltage = 4), 6 (joltage = 4), 14 (joltage = 7), and 15 (joltage = 8)).

In part 1 we were asked to select 2 batteries from each bank, and in part 2 we were to select 12. Here’s a solution:

function max_joltage_for_bank(bank::String; n_batteries::Int = 2)

    digits = parse.(Int, collect(bank)) 

    max_joltage = []; index = 1

    for k ∈ 1:n_batteries

        digits_needed = n_batteries - k
        search_end = length(digits) - digits_needed

        max_val = -1; max_idx = -1
        for i ∈ index:search_end
            if digits[i] > max_val
                # keep track of the biggest digit in this range and it's index...
                max_val = digits[i]; max_idx = i
            end
        end

        # ...so we can update the new starting point
        index = max_idx + 1
        push!(max_joltage, max_val)
    
    end

    return max_joltage |> join |> x -> parse(Int, x)
end

max_joltage_for_bank (generic function with 1 method)

This is based on selecting the maximum digit we can, whilst still leaving space for however many remaining batteries we need to select.

joltages |> x -> max_joltage_for_bank.(x, n_batteries = 12) |> sum

3121910778619

Don’t forget to then update your starting index accordingly, rather than just adding 1, otherwise you’ll keep selecting the same battery over and over - although seeing this helpded me fix it.

Check out some smarter solutions than mine:

-❄️- 2025 Day 3 Solutions -❄️-
byu/daggerdragon inadventofcode

⭐⭐

Day 4: Dunder Mifflin warehouse 📄

There are only so many types of AoC puzzles, and today I recognised bts of code I write every year. Namely, checking where I can safely move in in a grid while staying within bounds.

We’re given the layout of warehouse floor - specifically, where a reams of paper are located @, and where there is an empty space ..

grid = [
    "..@@.@@@@."
    "@@@.@.@.@@"
    "@@@@@.@.@@"
    "@.@@@@..@."
    "@@.@@@@.@@"
    ".@@@@@@@.@"
    ".@.@.@.@@@"
    "@.@@@.@@@@"
    ".@@@@@@@@."
    "@.@.@@@.@."
    ]

We are asked:

• part 1: If we can only move reams with less than 4 neighbouring reams - how many can we move?
• part 2: Once we’ve removed them, we might be able to move more. How many are left that can never be moved?

Here’s a solution:

function find_accessible(warehouse::Matrix{Char}; n_aux::Int = 4)

    n_rows, n_cols = size(warehouse); labelled_warehouse = copy(warehouse)
    
    # all 8 directions: (row_offset, col_offset)
    directions = [
        (-1, -1), (-1, 0), (-1, 1),
        ( 0, -1),          ( 0, 1),
        ( 1, -1), ( 1, 0), ( 1, 1)
    ]
    
    for row ∈ 1:n_rows
        for col ∈ 1:n_cols
            
            warehouse[row, col] != '@' && continue # we're only checking the papers
            
            neighbouring_papers = 0
            for (Δr, Δc) ∈ directions
                check_row, check_col = row + Δr, col + Δc
                
                if 1 ≤ check_row ≤ n_rows && 1 ≤ check_col ≤ n_cols # stay in the grid!
                    if warehouse[check_row, check_col] == '@'
                        neighbouring_papers += 1
                    end
                end
            end
            
            if neighbouring_papers < n_aux
                labelled_warehouse[row, col] = 'x'
            end
        end
    end
    
    return labelled_warehouse
end

find_accessible (generic function with 1 method)

grid |> 
    lines ->  [lines[r][c] for r ∈ eachindex(lines), c ∈ 1:length(lines[1])] |>
    warehouse -> count(==('x'), find_accessible(warehouse))

13

Not super elegeant. I replace the reams that can be moved with an x and then count them. It was important to make this change to a copy of the warehouse grid, otherwise we would be changing the grid as we go, and thinking there are less neighbouring reams than there actually are.

…and for part 2, we can just keep calling this function until there are no more reams that can be moved:

function remove_all_reams(grid::Matrix{Char})
    labelled_warehouse = copy(grid)
    while true
        new_labelled_warehouse = find_accessible(labelled_warehouse)
        new_labelled_warehouse == labelled_warehouse && break
        labelled_warehouse = new_labelled_warehouse
    end
    return count(==('x'), labelled_warehouse)
end

loevely visualisation on the reddit of clearing out the warehouse:

[2025 Day 4 Part 2]
byu/EverybodyCodes inadventofcode

⭐⭐

Day 5: fresh ingredients 🥗

Part 2 got me today.

Our inputs are a list of ranges, followed by ingredient id’s.

ingredients_db = [
    "3-5"
    "10-14"
    "16-20"
    "12-18"

    "1"
    "5"
    "8"
    "11"
    "17"
    "32"
]

An ingredient is fresh if its id falls within any of the ranges given. After a little pre-processing, we can collect all of the ranges that fresh ingredient fall into

…because we suspect we will need that for part 2, even though they end up going in a different, more complicated direction.

fresh_ingredients = Dict{Int, Vector{UnitRange{Int}}}()
for i ∈ ingredients
    for r ∈ ranges
        if i ∈ r
            push!(get!(fresh_ingredients, i, UnitRange{Int}[]), r)
        end
    end
end

and then returning the number of fresh ingredients.

fresh_ingredients |> length

3

Part 2 instead asks to find the number of possible fresh ingredients i.e. how many unique ingredient IDs fall within any of the given ranges.

…and in the real input data, the ranges were large, and overlapped. So, in cases where there was overlap, I combined them into a single new range (with the smaller of the start values, and larger of the end values).

sorted_ranges = sort(ranges, by = first)

fresh_ingredients_merged = [sorted_ranges[begin]]
for range ∈ sorted_ranges[2:end]
    # start a new range when the lower value is outside the largest value of the merge
    if last(fresh_ingredients_merged[end]) < first(range)
        push!(fresh_ingredients_merged, range)
    else
    # the start of the new range stays the same, but the end needs to be extended if it's bigger for the new range
    new_max_range = [last(fresh_ingredients_merged[end]), last(range)] |> maximum
    fresh_ingredients_merged[end] = first(fresh_ingredients_merged[end]):new_max_range
    end    
end

…and then summed the lengths of the now non-overlapping ranges (without duplicate ingredient IDs):

[length(x) for x ∈ fresh_ingredients_merged] |> sum

14

⭐⭐

Day 6: Cephalopod Maths 🐙

I googled cephalopod - it means a creature with a head and tentacles.

In the advent of code universe, they get maths homework, and this is what it looks like:

sums = [
    "123 328  51 64 ",
    " 45 64  387 23 ",
    "  6 98  215 314",
    "*   +   *   +  "
]

This is asking us to apply the operator in the last row, to the numbers above it. In part 1, the numbers above the first (leftmost) * are 123, 45, 6. In part 2 we are told the numbers are written in coluns (not rows) so we instead have to multiply 356, 24, and 1.

So the difference is in fiddling the parsing of the numbers. Here’s a solution:

function solve_sums(lines::Vector{String}; part_2::Bool = false)
    
    n_cols = lines[end] |> length
    # need to break until the next sum when there is a column of spaces
    no_sum = [all(line[c] == ' ' for line ∈ lines) for c ∈ 1:n_cols]

    results = Int[]; col = 1
    while col <= n_cols
        # skip the separator columns that split the sums
        while col <= n_cols && no_sum[col]
            col += 1
        end
        col > n_cols && break
        
        # otherwise we start a new sum
        start_sum = col
        
        # and keep going until the next separator
        while col <= n_cols && !no_sum[col]
            col += 1
        end

        # the operator is always in the first col of each sum, in the last row
        op = lines[end][start_sum]
        
        if part_2
            # pt.2: each column is a number (read top-down)
            numbers = Int[]
            for nums_col ∈ start_sum:col-1
                digits = [line[nums_col] for line ∈ lines[1:end-1] if line[nums_col] != ' ']
                if !isempty(digits)
                    push!(numbers, parse(Int, String(digits)))
                end 
            end
        else
            # pt.1: each row is a number, but they can have leading spaces
            numbers = [parse(Int, strip(line[start_sum:col-1])) for line ∈ lines[1:end-1]]
        end
        
        # an excuse to use Julia metaprogramming :)
        push!(results, op |> Symbol |> eval |> x -> reduce(x, numbers))
    end
        
    return results
end

solve_sums (generic function with 1 method)

Not completed on the day for the first time, but life is very hectic.

Oh, and an excuse to use Julia metaprogramming! We could have put in some conditional logic to take products when the operator is * and sums when it’s +, but instead we can directly convert to a symbol and evaluate it. Not at all required for this problem, but hey…

And then add up the results:

sums |> solve_sums |> sum

4277556

I’m inferring from the below meme, but I guess some people created matrices for all the numbers in each sum, rather than just strip the whitespace 🤔

Neat - wish I did it that way 🤷🏻

[2025 Day 6 (Part 2)] The solution be like…
byu/HotTop7260 inadventofcode

⭐⭐

Day 7: Tachyon beam splitting ⚛️

After yesterday, we’ve got 12 ⭐’s on the board

we’re 1/2 way there [Jon Bon Jovi voice]

A beam of tachyons, | flows from a starting point, S down a grid. When it hits a splitter, ^, two beams emerge (one either side).

tachyon_manifold = [
    ".......S.......",
    "...............",
    ".......^.......",
    "...............",
    "......^.^......",
    "...............",
    ".....^.^.^.....",
    "...............",
    "....^.^...^....",
    "...............",
    "...^.^...^.^...",
    "...............",
    "..^...^.....^..",
    "...............",
    ".^.^.^.^.^...^.",
    "..............."
]

In part 1, we are asked how many times the beam is split, and in part 2 we are asked how many parallel timelines (possible paths) exist for the beam.

Here’s my solution:

function analyse_beams(beam_rows::Vector{String})

    n_rows = beam_rows |> length
    n_cols = beam_rows[end] |> length
    grid = [beam_rows[row][col] for row ∈ 1:n_rows, col ∈ 1:n_cols]

    # find starting position
    start_pos = findfirst(==('S'), grid)
    start_row, start_col = start_pos[1], start_pos[2]

    # initialise beams and timelines...
    splits = 0; timelines = Dict(start_col => 1)
    
    for row ∈ (start_row + 1):n_rows
        # ...and then keep track of new ones
        new_timelines = Dict{Int, Int}()
        
        for (col, count) ∈ timelines
            if grid[row, col] == '^'
                # for part 1, we count the splits
                splits += 1 
                # for part 2, we "do" the split and count the timelines (add beams to the left and right - after an in-bounds check)
                col - 1 >= 1 && (new_timelines[col - 1] = get(new_timelines, col - 1, 0) + count)
                col + 1 <= n_cols && (new_timelines[col + 1] = get(new_timelines, col + 1, 0) + count)
            else
                new_timelines[col] = get(new_timelines, col, 0) + count
            end
        end

        timelines = new_timelines

    end

    return ("splits" => splits, "timelines" => timelines)
end

analyse_beams (generic function with 1 method)

This is a tidied up version, because I was initially separately tracking a variable beams, which I ended up separately recording in my part 2 dictionary. Also, I’ve got no idea if I needed the in-bounds checks (where I discount any beams that stray off the edges of the tachyon manifold).

tachyon_manifold |> analyse_beams

("splits" => 21, "timelines" => Dict(5 => 10, 13 => 1, 15 => 1, 7 => 11, 11 => 2, 9 => 11, 12 => 1, 3 => 2, 1 => 1))

Day 8: Connecting junction boxes 🔌

We are given 3-d (x, y, z) coordinates of electrical junction boxes and ask to connect the closest ones (shortest Euclidean distance) together to form circuits.

junction_boxes = [
    "162,817,812",
    "57,618,57",
    "906,360,560",
    "592,479,940",
    "352,342,300",
    "466,668,158",
    "542,29,236",
    "431,825,988",
    "739,650,466",
    "52,470,668",
    "216,146,977",
    "819,987,18",
    "117,168,530",
    "805,96,715",
    "346,949,466",
    "970,615,88",
    "941,993,340",
    "862,61,35",
    "984,92,344",
    "425,690,689"
]

The key insight: We don’t need to track every box in each circuit, just whether two boxes share a common “root”. This is a Union-Find (∪ + 🔎) problem.

Each box starts as it’s own standalone circuit. When we connect two boxes, one joins the other’s (the root’s) circuit. To check if two boxes are in the same circuit, we can then follow the chain and check if they share the same root.

coords = junction_boxes |>
    boxes -> split.(boxes, ",") |>
    boxes -> [parse.(Int, b) for b ∈ boxes]

# each box starts as its own root
n_boxes = length(coords); parent = collect(1:n_boxes)

# follow the chain upward to find the root
function find_circuit_root(box_index)
    while parent[box_index] != box_index
        box_index = parent[box_index]
    end
    return box_index
end

# connect two circuits by making one root point to the other
function connect_boxes!(box_from, box_to)
    root_from = find_circuit_root(box_from)
    root_to = find_circuit_root(box_to)
    
    # already in same circuit?
    root_from == root_to && return false
    
    # otherwise => merge
    parent[root_from] = root_to
    return true
end

Here I check the distance between every pair of boxes, and arrange shortest to longest:

distances = []
for box ∈ 1:n_boxes
    for alt ∈ (box+1):n_boxes  # avoid double-counting pairs
        dist = (coords[box] .- coords[alt]).^2 |> sum |> sqrt
        push!(distances, (dist = dist, from = box, to = alt))
    end
end
sort!(distances, by = first)

Part 1 asks us to attempt to connect the 1000 shortest connections and part 2 asks wants us to keep going until we end up with 1 big circuit and a final connection. Here’s my solution:

function size_circuits(distances; n_attempts::Union{Int, Nothing} = nothing)
    
    # each box starts as is its own circuit
    parent = collect(1:n_boxes)
    last_connection = nothing

    for (attempt, d) ∈ enumerate(distances)
        
        if connect_boxes!(d.from, d.to)
            last_connection = d
        end

        # part 1: stop after N attempts
        if !isnothing(n_attempts) && attempt >= n_attempts
            break
        end
        
        # part 2: stop when everything is connected
        roots = [find_circuit_root(i) for i ∈ 1:n_boxes]
        length(unique(roots)) == 1 && break
    end

    roots = [find_circuit_root(i) for i ∈ 1:n_boxes]
    circuit_sizes = [count(==(r), roots) for r ∈ unique(roots)]
    sort!(circuit_sizes, rev=true)

    return Dict("circuit_sizes" => circuit_sizes, 
                "last_connection" => last_connection)
end

size_circuits (generic function with 1 method)

There are some lovely visualisations of the growing mega-circuit of part 2:

[2025 Day 8 (Part 2)] Visualisation
byu/Derailed_Dash inadventofcode

size_circuits(distances)["last_connection"] |>
    last -> coords[last.from][1] * coords[last.to][1]

25272

The puzzles are getting tricky. This is possibly the first one that can’t be brute-forced? My naive approach of adding new boxes to sets of circuits was pretty slow (O(n)). This is also the first puzzle I was unable to solve them on the day of release due to other commitments, but I’m not fussed about that.

⭐⭐

Day 9: Red and Green tiles 🟥 🟩

We are provided with coordinates (in 2-d today) of red tiles:

red_tiles = [
    "7,1",
    "11,1",
    "11,7",
    "9,7",
    "9,5",
    "2,5",
    "2,3",
    "7,3"
]

We are asked to find the largest (by area) rectangle that can be made, if we took two red tiles as opposite corners. Part 1 didn’t have any further restrictions and so was one of the simplest stars to get so far this year. Part 2 was not…

In part 2, we connect sequential red tiles using green tiles. We then have a border of green tiles since, adjacent tiles are always in the same row or column (thankfully!) and we also fill in the inside of this shape with green tiles. The large rectangles we are trying to form cannot include any coordinates that are not either red or green tiles.

How do we know which tiles are “inside” the loop?

I think there are some clever ways to do this by flooding the outer grid with empty tiles or the inner grid with green tiles, until we hit the border.

I used a simpler approach. For each empty tile, walk left and count how many times you cross vertical boundaries. After an odd number of crossings, you must be inside (green tile). 🪄 Trick: be carefull to ignore horizontal boundaries, as they will just run parallel to our walk (we won’t cross them). This is why I added the above check.

# connect sequential red tiles with green tiles
function draw_green_line!(grid, from, to)
    x_min, x_max = minmax(from[1], to[1])
    y_min, y_max = minmax(from[2], to[2])
    
    for x ∈ x_min:x_max
        for y ∈ y_min:y_max
            grid[x, y] == 0 && (grid[x, y] = 2)  # only fill empties
        end
    end
end

draw_green_line! (generic function with 1 method)

# are we inside the shape?
function is_inside(grid, x, y)
    grid[x, y] != 0 && return false  # already filled
    
    crossings = 0
    for check_x ∈ 1:(x-1)
        # counting vertical boundaries
        at_y = grid[check_x, y] ∈ (1, 2)
        above = y > 1 && grid[check_x, y-1] ∈ (1, 2)
        at_y && above && (crossings += 1)
    end
    
    return isodd(crossings)
end

is_inside (generic function with 1 method)

Turns out the grid is enormous - 9655610113 tiles.

Since we only care about red tiles, we can index their sparse coordinates. I eventually remembered to use the original (enormous) grid coordinates to calculate how big the rectangle areas are. Here’s my solution:

# check rectangle contains only red/green tiles
function is_valid_rectangle(grid, c1, c2)
    x_min, x_max = minmax(c1[1], c2[1])
    y_min, y_max = minmax(c1[2], c2[2])
    
    for x ∈ x_min:x_max
        for y ∈ y_min:y_max
            grid[x, y] == 0 && return false
        end
    end
    return true
end

is_valid_rectangle (generic function with 1 method)

function max_area(red_tiles_coords::Vector{Vector{Int}}; part_2::Bool = false)
    
    n_tiles = length(red_tiles_coords)
    max_area = 0
    best_pair = nothing

    if part_2
    # coordinate compression
        all_x = sort(unique(c[1] for c ∈ red_tiles_coords))
        all_y = sort(unique(c[2] for c ∈ red_tiles_coords))
        
        # big grid => compressed index
        x_to_idx = Dict(x => i for (i, x) ∈ enumerate(all_x))
        y_to_idx = Dict(y => i for (i, y) ∈ enumerate(all_y))
        
        # Compress all red tile coordinates
        compressed_coords = [[x_to_idx[c[1]], y_to_idx[c[2]]] for c ∈ red_tiles_coords]
        
        # using reduced grid dims...
        max_x = length(all_x)
        max_y = length(all_y)
        grid = zeros(Int, max_x, max_y)
        
        # ...mark red tiles and connect with greens
        for c ∈ compressed_coords
            grid[c[1], c[2]] = 1
        end
        
        for i ∈ 1:n_tiles
            next_i = i == n_tiles ? 1 : i + 1
            draw_green_line!(grid, compressed_coords[i], compressed_coords[next_i])
        end
        
        # and paint the interior green too
        for x ∈ 1:max_x
            for y ∈ 1:max_y
                is_inside(grid, x, y) && (grid[x, y] = 2)
            end
        end
    end

    for i ∈ 1:n_tiles
        for j ∈ (i+1):n_tiles
            
            # part 2: check validity using compressed coords
            if part_2 && !is_valid_rectangle(grid, compressed_coords[i], compressed_coords[j])
                continue
            end

            # ...but then calculate the area on the big grid
            width = abs(red_tiles_coords[i][1] - red_tiles_coords[j][1]) + 1
            height = abs(red_tiles_coords[i][2] - red_tiles_coords[j][2]) + 1
                        
            if width * height > max_area
                max_area = width * height
                best_pair = (i, j)
            end
        end
    end

    return max_area
end

max_area (generic function with 1 method)

which matches the example answer:

red_tiles |>
    tiles -> split.(tiles, ",") |>
    tiles -> [parse.(Int, t) for t ∈ tiles] |> 
    coords -> max_area(coords, part_2 = true)

24

A hodgepodge of a solution, which may have some breaking edge cases, but got me there. This is the first time I have been 3 puzzles away from completing advent of code…

⭐⭐

Day 10: ⏳

…

autodiff (AD)

So much of computational statistics and machine learning is built on automatic (algorithmic) differentiation - AD, “autodiff”, or “autograd”. Dig into literature on Bayesian inference, or deep learning, and you will find parameters being nudged in a direction informed by a gradient. I’m sometimes surprised at the extent of the ‘gradient-based’ monopoly in scientific computing, but I don’t mean to trivialise! …getting gradients of complex functions, very quickly and without error, is a powerful tool and its great that we are able to leverage this.

AD works by “the relentless application of the chain rule”, as I vaguely recall one of the Stan developers saying. Large functions are differentiated piece by piece (for which look-up rules can be applied), and the results are stitched together.

inter-operability and Julia ♥️💚💜

Perhaps my favourite feature of Julia is its inter-operability. Look at a GitHub repo for a Julia package and you will generally find the following:

Julia is so performant, that its libraries for scientific computing will work with normal variables - without needing to package up their own types. So solvers from DifferentialEquations, or neural networks defined in Flux can be immediately combined with the Julia probabilistic programming language, Turing.

If a new framework emerges in Python, an entirely new ecosystem may need to be developed to prop it up. This may involve duplicating existing but now incompatible functionality - think of JAX needing to implement its own NumPy module. …whereas if you write a new Julia library, it could offer a vast range of applications as it is combined with other packages, which feels like potential for a multiplicative, rather than additive, impact.

Here I am, impressed that the interoperability of the Copulas package:

pic.twitter.com/ySAg1SM98g

— Domenic Denicola (@Domenic_DF) January 31, 2023

AD in Julia

ML frameworks in Python require you to work in their own syntax, with their specific types, and use their own in-built AD methods. Locking into a framework is not ideal, as you are limited to the methods they support and you need to juggle types of inputs and outputs.

Conversely, Julia works the other way around. You write your code, using whichever libraries, functions and types you want, and then you choose an AD library to get you the gradients you need.

Why does this feel so powerful? its the promise of gradient-based methods for your scientific problem, rather than a walled-off machine learning model. You can point an AD engine at the aspect of your analysis that you are interested in for more bespoke interrogation or optimisation.

I recently completed a project, which had an element of adversarial AI (counterfactual analysis and saliency maps), which I wrote in JAX, mainly as an excuse to learn JAX. I used their NNX module for neural networks, and found this to be unexpectedly restrictive. I wasn’t able to run analysis that required gradients of outputs w.r.t. inputs, as I hit NNX errors/limitations that I wasn’t able to resolve. I ended up re-writing everything in Julia.

Once upon a time there was Zygote: an AD library that powered ML in Julia. A great achievement, but limitations began to emerge. Because of how it operates on your code, it can struggle with certain features. This is similar to the so-called “sharp bits” of JAX …though those limitations emerge for very different reasons.

Zygote is also sometimes described as “too permissive”. This is because it intercepts your code at some intermediate representation level (see note below), performs AD and then reassembles. Julia’s IR levels were designed for performance, not AD and so doesn’t reliably detect and report errors in Zygote’s process.

It is possible for AD solutions to work with the compiler, rather than against it — more on this in Part 2! But for now…

# some nice high-level code
nice_function(x) = 2x + 1

nice_function (generic function with 1 method)

# the "lowered" IR
@code_lowered nice_function(1.0)

CodeInfo(
1 ─ %1 = Main.:+
│   %2 = Main.:*
│   %3 = (%2)(2, x)
│   %4 = (%1)(%3, 1)
└──      return %4
)

# the "typed" IR
@code_typed nice_function(1.0)

CodeInfo(
1 ─ %1 = Base.mul_float(2.0, x)::Float64
│   %2 = Base.add_float(%1, 1.0)::Float64
└──      return %2
) => Float64

# the LLVM level
@code_llvm nice_function(1.0)

; Function Signature: nice_function(Float64)
;  @ none:3 within `nice_function`
define double @julia_nice_function_7614(double %"x::Float64") #0 {
top:
;  @ none:5 within `nice_function`
; ┌ @ promotion.jl:430 within `*` @ float.jl:493
   %0 = fmul double %"x::Float64", 2.000000e+00
; └
; ┌ @ promotion.jl:429 within `+` @ float.jl:491
   %1 = fadd double %0, 1.000000e+00
   ret double %1
; └
}

# finally, actual machine code
@code_native nice_function(1.0)

    .section    __TEXT,__text,regular,pure_instructions
    .build_version macos, 15, 0
    .globl  _julia_nice_function_7864       ; -- Begin function julia_nice_function_7864
    .p2align    2
_julia_nice_function_7864:              ; @julia_nice_function_7864
; Function Signature: nice_function(Float64)
; ┌ @ none:3 within `nice_function`
; %bb.0:                                ; %top
; │ @ none within `nice_function`
    ;DEBUG_VALUE: nice_function:x <- $d0
    ;DEBUG_VALUE: nice_function:x <- $d0
; │ @ none:5 within `nice_function`
; │┌ @ promotion.jl:430 within `*` @ float.jl:493
    fadd    d0, d0, d0
    fmov    d1, #1.00000000
; │└
; │┌ @ promotion.jl:429 within `+` @ float.jl:491
    fadd    d0, d0, d1
    ret
; └└
                                        ; -- End function
    .section    __DATA,__const
    .p2align    3, 0x0                          ; @"+Core.Float64#7866"
"l_+Core.Float64#7866":
    .quad   "l_+Core.Float64#7866.jit"

.set "l_+Core.Float64#7866.jit", 6237349104
.subsections_via_symbols

And then came Enzyme: an AD library that works at the LLVM level (your code is compiled first, and then the gradients are computed). This led to improvements in both performance and flexibility - we can now differentiate through mutation and control flow. This more resilient library has a steeper learning curve (imo), but (also imo) requiring more explicit instructions ends up making things clearer.

I hope the below examples will help get you started.

example 1: a Bayesian linear regression

How about computing the gradients we need for Hamiltonian Monte Carlo sampling. For a linear regression model, with inputs, and outputs, :

with priors:

Let’s put these in a tuple:

using Distributions

priors = (
    α_prior = Normal(0, 3), 
    β_prior = Normal(0, 3), 
    σ_prior = Exponential(1)
)

(α_prior = Normal{Float64}(μ=0.0, σ=3.0), β_prior = Normal{Float64}(μ=0.0, σ=3.0), σ_prior = Exponential{Float64}(θ=1.0))

For numerical stability reasons MCMC typically works in negative log space, so the below function finds the unnormalised negative log posterior for our model. This could of course be sped-up, but I’ve tried to keep it friendly 😊

using Random

function neg_log_posterior(params::NamedTuple, priors::NamedTuple,
                          x::Vector, y::Vector)

    α = params.α; β = params.β; σ = params.σ
    
    # mean of likelihood
    μ_pred = α .+ x * β
    
    # increment the negative log likelihood for all observations
    neg_log_lik = 0.0
    for i in 1:length(y)
        neg_log_lik += -logpdf(Normal(μ_pred[i], σ), y[i])
    end
    
    # ...and for the priors
    neg_log_prior_α = -logpdf(priors.α_prior, α)
    neg_log_prior_β = -logpdf(priors.β_prior, β)
    neg_log_prior_σ = -logpdf(priors.σ_prior, σ)
    
    # summing in log space is equivalent to multiplying priors and likelihoods 😉
    return neg_log_lik + neg_log_prior_α + neg_log_prior_β + neg_log_prior_σ
end

neg_log_posterior (generic function with 1 method)

To use this function, we need to define some inputs. Here, I’m just simulating some data, using “true” parameter values:

# 20 data points, why not...
n_samples = 20

# define a PRNG for reproducibility
prng = MersenneTwister(231123)

# inputs from a standard Gaussian
x = randn(prng, n_samples)

# outputs by sending inputs through a "true" model
α_true = 1/2; β_true = -1/2; σ_true = 2

y = α_true .+ x * β_true .+ σ_true * randn(prng, n_samples)

We can use Enzyme to get the gradients of the negative log posterior w.r.t. the model parameters - as required by Hamiltonian Monte Carlo.

# where do i want gradients?
params_init = (
    α = rand(prng, priors.α_prior),
    β = rand(prng, priors.β_prior),
    σ = rand(prng, priors.σ_prior)
)

(α = -6.290988779313054, β = 3.9183834605694834, σ = 1.6155587713748865)

I am giving the gradient() function three arguments:

• the mode/direction to apply AD, Reverse. Each pass of a reverse-mode AD computes gradients of all inputs w.r.t. a single output (as a vector-Jacobian product). In Forward mode, each pass computes gradients of a single input w.r.t. all outputs (as a Jacobian-vector product).

  See above callout note for more on this and why reverse mode is likely not an optimal choice for so few parameters.

  Consequently, there are efficiency trade-offs associated with this selection, depending on the number of inputs and outputs of…

• …the function we are differentiating, params -> neg_log_posterior(params, priors, x, y). Here, an anonymous function that takes params as input and returns the negative log posterior, using neg_log_posterior(), which we defined above.

• the point at which we want gradients, params_init. This is the current location of the Markov chain. In the first instance we need an initial guess, for which we have drawn from the priors.

using Enzyme
# computing the gradients
∇θ = Enzyme.gradient(
    Reverse, 
    params -> neg_log_posterior(params, priors, x, y), 
    params_init
)

((α = -54.13440425830732, β = 25.990919631591435, σ = -323.2246379556173),)

We can then use these gradients to update the momentum of our Hamiltonian ‘particles’, generating proposals guided by the geometry of the posterior distribution. Unlike random walks or Gibbs samplers, this generation of samplers remain efficient in high dimensions 🥳

example 2: an MLP (simple neural network)

I defined a simple, densely connected neural network without anything clever (no layer normalisations, recurrent connections or attention mechanisms), sometimes referred to as a multi-layer perceptron (MLP).

I’ll spare you this set-up code here as we are focussing on autodiff, but you can find the full code on GitHub.

Instead, let’s look at my training function. Notice that I am now using a different function, Enzyme.autodiff() for backpropagation. It has more arguments:

• the mode/direction to apply AD, set_runtime_activity(Reverse). Similar to gradient(), but here we are specifying that we want to use reverse-mode AD, with runtime activity analysis. As a rule of 👍, I start with regular Reverse mode AD. If I get compilation errors about, for instance, type inference or broadcasting, then I add set_runtime_activity().
• the function we are differentiating, (net, funs, inputs, targets) -> find_loss(net, funs, inputs, targets). Here, an anonymous function that takes the neural network, its functions, inputs and targets as arguments, and returns the loss.
• the activity of the function Active. we need to make the output to the loss function active, because it is the starting point of the chain rule.
• the activity of each argument, Active, Const(), or Duplicated(). This is where things get more explicit. We need to tell Enzyme which arguments we want gradients for (Active), and which we don’t (Const) - the derivative of a constant is zero. Finally, we also want gradients for Duplicated variables, but they could be large. So we create a shadow copy of the neural network, nn_shadow, which we use to accumulate gradients in-place (without allocating new memory each time!)

function train(nn::neural_network, nn_funs::neural_network_funs, 
               a::Array{Float64}, y::Array{Float64};
               a_test::Array{Float64} = a, y_test::Array{Float64} = y, 
               n_epochs::Int = 10, η::Float64 = 0.01)
    @assert n_epochs > 0 "n_epochs must be greater than 0"

    training_df = DataFrame(epoch = Int[], loss = Float64[], test_loss = Float64[])

    for i in 1:n_epochs

        # initiate our memory-saving shadow
        ∇nn = Enzyme.make_zero(nn)

        # find ∂ℒ/∂θ
        Enzyme.autodiff(
            set_runtime_activity(Reverse),
            (net, funs, inputs, targets) -> find_loss(net, funs, inputs, targets)[1],
            Active,
            Duplicated(nn, ∇nn),
            Const(nn_funs),
            Const(a),
            Const(y)
        )

        # nudge all weights and biases towards a lower loss, using learning rate, η
        for j = 1:length(nn.Ws)
            nn.Ws[j] -= η * ∇nn.Ws[j]
            nn.bs[j] -= η * ∇nn.bs[j]
        end
        
        # record losses
        append!(training_df, 
                DataFrame(epoch = i, 
                          loss = find_loss(nn, nn_funs, a, y)[1],
                          test_loss = find_loss(nn, nn_funs, a_test, y_test)[1]))

    end
    
    return nn, training_df
end

Enzyme.make_zero(nn) creates a structural copy of nn with the same type (neural_network), field names (Ws, bs), and dimensions …but with all numerical values set to zero. This memory-saving trick is important for large vectors of parameters, as we will generally have in deep learning.

The example applications that I selected are already very well equipped with sophisticated Julia libraries. If you are interested in probabilistic modelling in Julia, use Turing, if you are interested in deep learning, use Flux. Both are Enzyme compatible, but the later has specific guidance on how to set this up, using the Duplicated method that we used above:

some references

The Julia autodiff ecosystem, which is more vast than the examples covered in this blog post, link

A summary of the key trade-offs accross various autodiff methods, link

Professor Simone Scardapone’s book, “Alice’s adventures in a differential wonderland” link.

“As the name differentiable implies, gradients play a pivotal role”

JuliaCon talk on Julia’s unique approach to autodiff:

Bayes@Lund2023 Conference

I have been following the Bayes@Lund conference since I started my PhD, and have often found the work presented to be very useful. This year I was able to attend and I presented on the topic of value of information analysis (how much should we be willing to pay for data).

Conventional experimental design is used to identify where our next mesaurement(s) should be obtained on the bases of reducing uncertainty. However, this scale (some measure of information entropy) is not always intuitive, and it won’t tell you the point at which paying for another measurement becomes uneconomical.

Value of information analysis is used to quantify how much we should be willing to pay for data of a specified quality (precision, bias, reliability, completeness, etc.), in the context of helping us make decisions.

Below is the recording of my talk, which breifly introduces the topic and provides a couple of examples.

A (Questionable) Measure of Player Form

As I alluded to in the TLDR above, I suspect a players form is somehow linked to their current mental state. When they are feeling confident they may be less likely to doubt their abilities, and more decisive. This could mean they act quicker and become more difficult to play against.

I will not propose a detailed causal model here, just a statistical proxy. But, I will be assuming that form can rise and decay over time. For some players even a single good or bad performance may be enough to drastically impact their next game, and for others this process may be smoother and less volatile. More on this later.

For the purposes of this post, goalscoring form on a given match day, , is defined as the difference between the number of goals that were scored on that match day, minus the expected number of goals, associated with the opportunities in that game.

A nice feature of this is that, in principle, it is invariant to the quality of opposition. A striker may have a higher xG when performing against a weaker team, but will therefore need to score more goals in such a game to be considered in the same form. By the same token, it should also account for the fact that a player will generally get fewer scoring opportunities as a substitute.

…As for the not so nice features, there are plenty! For instance, what good is a measure of form that only considers goals scored? Is xG not also conditional on how well a striker is paying? Would it be more useful to standardise the result?

These are all fair questions, and with a little thought could all be integrated into a more comprehensive characterisation. However, the type of model that I will introduce will be equally compatible with alternative definitions, so let’s imagine we just care about whether a striker is scoring as many goals as they should be, and whether this will continue.

Ellen White

Ellen White is a clinical striker who, at the time of writing this, plays for Manchester City and England. She is a former winner of the Women’s Super League (WSL) golden boot, and is England’s all-time top scorer. So plenty of opportunities to see her distinctive celebration:

StatsBomb have kindly made data from the WSL (2019/20 season) freely available in their R package, and so we will consider this league season of Ellen White’s career here.

Here is a plot of Ellen’s form (performance vs. xG) over the 12 league games that she featured in, during that season. Her biggest over performance vs. xG was when she scored in a 4-1 win away at Tottenham despite a cumulative xG of 0.597. Her worst performance by the same measure was the following week, failing to score in the 3-1 win at home to Everton. Although she only played the final minutes of this game, she accrued a match xG of 0.302. The fact that these games were back to back could be tricky for a form model to accommodate!

What would such a model need to do?

Essentially, we are looking for are some numbers to help us understand the following:

• What form is a player currently in?
• How long will a player remain in good (or bad) form?
• What is the uncertainty in our predictions?

Since players can enter good and bad patches of form over the course of a season, we need a model that is able to twist and turn accordingly. This means we need some non-linearity.

We also want future predictions to be based on recent games - if a player has over performed for the last 3 games in a row, then we generally expect them to continue on this path, at least in the short-term. But how far back should we look? Does a single great performance from months ago have any impact on a players current form? We need to quantify this dependency in our model too.

Finally, a probabilistic model has the benefit of quantifying uncertainty. I emphasise the importance of this in the ‘final thoughts’ at the end of this post and for anyone interested, here is more Bayesian statistics propaganda. But, in summary we should not neglect uncertainty in this model because (a) We are not even sure what form is, and (b) we are estimating it from a small amount of indirect observations. So let’s not pretend we will end up with a single number. Enough preaching and back to the task at hand….

One solution that checks the above requirements is the Gaussian Process (GP).

What does it look like?

So we have this probabilistic model of smooth, non-linear functions. Let’s see what it looks like. In the below plot, the match days are the same as those presented in the above plot, we just have a new y-axis scale, and we have ‘days’ (rather than date) along the x-axis.

There are multiple functions that are consistent with Ellen White’s form in the league that season, so let’s look at one example first:

Where there is a large gap between successive games (such as the 3 weeks between White’s first and second appearances of the season), there is less evidence to guide predictions of form. This is also true for the period around day 60 (late December). Here though, she was on an upward trajectory. In both cases, this lack of data results in higher uncertainty, as is apparent when we look at more samples, which are shown on top of the full predictive distribution below.

And, so what?

Some squiggly lines that approximately go through some points? What is the value of this when you could scribble something similar without knowing anything about statistics?

Well, underlying all of these lines is a model of dependency. We have quantified how similar (correlated) form should be in successive games, and how this correlation will decrease with time. I will talk about the parameters that do this and how they can be interpreted in part 2 (in preparation), but to summarise, the model quantifies how correlation in form decreases as time progresses and this can be seen in the smoothness of the lines.

For example, if form was always shown to be very similar to that of the previous game, then transitioning from good to bad performances would be gradual, and the samples from the associated GP model would be very smooth. Conversely, in the case where performance in subsequent games were completely independent, even if very little time had passed, the GP regression lines would need to be able to change direction very sharply.

Below are some predictions from the model in the period just after White’s last game of the season (to the right of the final match day on the above plots). She appeared to be on a slight upward trend at this point, over performing in her final game at home to Chelsea. This is shown in the uppermost histogram. As we move away from this game, into the off-season, we see the uncertainty gradually increase in our predictions and the average move towards zero. This is consistent with the considerations discussed above.

Some final thoughts…

Expected Goals

Expected Goals (or xG) is a metric that was developed to predict the probability of a football (soccer) player scoring a goal, conditional on some mathematical characterisation of the shooting opportunity. Since we have a binary outcome (he or she will either score or not score) we can use everyone’s favourite GLM - logistic regression.

Unfortunately this causes some overlap with a previous blog post - ‘Bayesian Logistic Regression with Stan’, but don’t worry - the focus here is all about Partial Pooling.

First let’s look at a non-Bayesian base case. StatsBomb have kindly made lots of football data freely available in their R package. The below creates a dataframe of the shots taken by Arsenal FC during the 2003-04 Premier League winning season.

library(StatsBombR); library(tidyverse)

Prem_SB_matches <- FreeMatches(Competitions = SB_comps %>% 
                               dplyr::filter(competition_name == 'Premier League') %>% 
                               dplyr::filter(competition_gender == 'male'))

Arsenal_0304_shots <- StatsBombFreeEvents(MatchesDF = Prem_SB_matches, 
                                          Parallel = TRUE) %>% 
  allclean() %>% 
  dplyr::filter(type.name == 'Shot') %>% 
  dplyr::filter(possession_team.name == 'Arsenal')

Using R’s tidymodels framework - make sure to have a look at Julia Silge’s tutorials if you are unfamiliar - we can specify and fit a logistic regression. The below compares our results (including confidence intervals) to those from StatsBomb.

If you are interested in creating something similar yourself, this model has standardised inputs for parameters with relatively large values (such as angles and distances) and one hot encoding of categorical inputs (such as whether or not the shot was taken with a players weaker foot).

Since we have used StatsBomb data (though their model will no doubt be based on a much larger collection) we would expect our results to be similar to theirs, and they are. Considering just the point estimates, the two models appear to broadly agree, especially when both are predicting a very low or a very high xG.

However, some of the confidence intervals on our tidymodels predictions are very large. Although we would generally expect these to decrease as we introduced more data, we know that football matches (and especially specific events within football matches) are full of uncertainty. If we want to be able to quantify this uncertainty in a more useful way (we do) - we want a Bayesian model. The below section details the specific type of Bayesian model that I’m proposing for estimating xG.

Multi-Level (Partial Pooling) Models

Hierarchical (or ‘nested’) data contains multiple groups within a population, such as players with a football team. Unfortunately, this information is lost (and bias is introduced) when such data is modelled as a single population. At the other extreme we can assume each group is fully independent, and the difficulty here is that there will be less data available and therefore more variance in our predictions.

Consequently, we want an intermediate solution, acknowledging variation between groups, but allowing for data from one group to inform predictions about others. This is achieved by using a multi-level (or hierarchical) model structure. Such models allow partial sharing (or pooling) of information between groups, to the extent that the data indicate is appropriate. This approach results in reduced variance (when compared to a set of corresponding independent models), a shift towards a population mean (known as shrinkage), and generally an improved predictive performance.

Sounds great, right? So why would anyone ever not use this kind of model? In his excellent blog, Richard McElreath makes the case that multi-level models should be our default approach. His greatest criticism of them is that they require some experience or training to specify and interpret. His book has a dedicated chapter to help with that. Of course, there are many better descriptions of multi-level modelling than you will get from me, but I personally found the examples in Andrew Gelman and Jennifer Hill’s book to be very helpful. Finally, Michael Betancourt has written a much more comprehensive blog post on the topic, which includes a discussion on the underlying assumption of exchangeability.

We can create a partial pooling model by re-writing the below:

To look like this:

In this new structure, each parameter will now be a vector of length (where players are being considered). This means there will be a different co-efficient describing how varies with distance from goal for each player. This makes sense as we would expect variation between players and we want our model to be able to describe it.

If each of these parameters had their own priors, we would essentially have specified independent models - one for each player. But there is a twist here: each of the vectors of co-efficients share a single prior.

This will pull each of the individual co-efficients towards a shared mean, . The variation between the players (for a given parameter) is characterised by . Rather than specify these ourselves, we will also estimate these as part of the model. This means that the extent of the pooling is conditional on the data, which is an extremely useful feature. However, we then need to include priors on these parameters, which are known as hyperpriors.

Note that this process has introduced an extra layer (or level) to the model structure. This is why they are known as multi-level or hierarchical models. The term partial pooling is more a description of what they do.

We see the greatest benefit of this approach when only limited data is available for one or more groups. If one player took very few shots during a period of data collection, then there will be a lot of uncertainty in their xG predictions ….unless we can make use of the data we have for the rest of the team.

What does this look like in Stan?

The below is a reduced Stan model, with just one co-efficient (concerning the distance from goal of the shot). This is not me being secretive, its just that the full model is quite large. You can simply add more parameters like a multi-variate linear regression on the log-odds scale, but remember that they will each require priors, hyperpriors, and data.

data {

  int <lower = 1> n_shots;
  int <lower = 0, upper = 1> goal [n_shots];
  
  int <lower = 1> n_players;
  int <lower = 1> player_id [n_shots];
  
  vector [n_shots] dist_goal;

  real mu_mu_alpha;
  real <lower = 0> sigma_mu_alpha;
  real<lower = 0> rate_sigma_alpha;
  
  real mu_mu_beta_dist_goal;
  real <lower = 0> sigma_mu_beta_dist_goal;
  real<lower = 0> rate_sigma_beta_dist_goal;
  
}

parameters {
  
  vector [n_players] alpha;
  
  vector [n_players] beta_dist_goal;

  real mu_alpha;
  real <lower = 0> sigma_alpha;
  
  real mu_beta_dist_goal;
  real <lower = 0> sigma_beta_dist_goal;
  
}

model {

  // Logistic model 
  
  goal ~ bernoulli_logit(alpha[player_id] + beta_dist_goal[player_id] .* dist_goal); 

  // Priors 
  
  alpha ~ normal(mu_alpha, sigma_alpha);
  beta_dist_goal ~ normal(mu_beta_dist_goal, sigma_beta_dist_goal);

  // Hyperpriors
  
  mu_alpha ~ normal(mu_mu_alpha, sigma_mu_alpha);
  sigma_alpha ~ exponential(rate_sigma_alpha);
  
  mu_beta_dist_goal ~ normal(mu_mu_beta_dist_goal, sigma_mu_beta_dist_goal);
  sigma_beta_dist_goal ~ exponential(rate_sigma_beta_dist_goal);
  
}

generated quantities {
  
  real alpha_pp = normal_rng(mu_alpha, sigma_alpha);
  real beta_dist_goal_pp = normal_rng(mu_beta_dist_goal, sigma_beta_dist_goal);

}

A few things that I’d like to note:

• My input data is of length n_shots and my parameters are vectors of length n_players.
• I’ve included my hyperpriors (the mu_mu_..., sigma_mu..., and rate_sigma... terms) as data, rather than hard code values into the file. This is so I can re-run the model with new hyperpriors without Stan needing to recompile.
• Even though I have included the mu... and sigma.. terms as priors in my comment, this is just to help describe the model structure. They are all included in the Parameters block of the model. As discussed above, they are inferred as part of the joint posterior distribution, meaning that we are estimating the extent of the pooling from the data.
• I’m using the generated quantities to produce my population-level parameters, so that I have everything I need to put together probabilistic predictions in either R or Python.

#### Model Parameters

The posterior distribution (which Stan has sampled from) is a joint probabilistic model of all parameters. Let’s have a look at a few, specifically those corresponding to the effect of distance between the shot taker and goalkeeper. Shown below is the co-efficient for players (indexed ). We can see that the distance to the keeper is predicted to influence each player differently.

Some of the players will have taken fewer shots and therefore we will have less data to fit their player-specific parameters. The mu_beta_dist_keeper and sigma_beta_dist_keeper parameters in the above plot are the shared ‘priors’ that control how the data from each of the players can be used to inform one another. The beta_dist_keeper_pp parameter is specified in the generated quantities block of my Stan model. It is correlated samples from the distribution characterised by the shared priors. This becomes the population (team) level co-efficient in my predictions.

I’ve included some predictions for some actual shots taken that season in part 1 of this article, but since this is the purpose of the model let’s look at one more.

Here is Robert Pirès goal from just outside the box at home to Bolton Wanderers in 2004. It was on his stronger (right) foot and he was not under pressure from any defenders.

As labelled on the above plot, the StatsBomb model only gave Pirès a 5% chance of scoring this chance. The below xG predictions are from the Bayesian partial pooling model, both for Robert Pirès (upper) and for the case where any Arsenal player could be shooting (lower). Also shown is the StatsBomb prediction. We see an improvement (since we know this chance was scored) when we make a player-specific prediction.

Our probabilistic predictions contain more information than point estimates, but for the purposes of a simpler comparison we can consider the mean value. The mean value of our team-level prediction is 20%, but conditional on the knowledge that Pirès was shooting, this becomes 33%.

If Arsène Wenger could’ve chosen which of his players was presented with this opportunity, Robert Pirès would’ve been one of his top choices (though possible behind Thierry Henry). We have an intuitive understanding that such players have the necessary attributes to score from relatively difficult opportunities such as this, and this is accounted for in our model. We have tackled the challenge of greatly reduced (player-specific) datasets, by allowing them to share information on the basis of how similar they are.

Challenges

Multi-level models capture the multi-level structure of hierarchical (nested) datasets, accounting for both variability and commonality between different groups (in this example: between different players in a team). However, as we can see from the previous plot, by introducing a set of parameters for each group and relating them all in this way, the posterior distribution now has many more dimensions and is more challenging to sample from. If you are using Stan you may now see more warning messages regarding divergent transitions - a concept that José Mourinho is acting out, below. If you do run into these problems, I would recommend reviewing the guidance in the Stan manual on reparameterisation (writing your same model on a new scale, such that it is easier for the software to work with).

Finally, I have published a paper demonstrating this modelling approach in an engineering context, which includes additional details for anyone who is interested: ‘Consistent and coherent treatment of uncertainties and dependencies in fatigue crack growth calculations using multi-level Bayesian models’.

What are Expected Goals?

Opta sports tell us that the Expected Goals (or xG) of a shot describe how likely it is to be scored. The cumulative xG over a game will therefore give an indication of how many goals a team would usually score, based on the chances they created.

Why would anyone be interested in this? Because if the xG model is any good, it can be the basis for an evidence-based style of play. If certain individuals in a team enjoy shooting from long-distance (or any other set of circumstances associated with a low xG), they may be encouraged to keep possession until a more favourable (higher xG) chance arises.

There is no universally accepted way of calculating xG, so there are many competing models around. In this article I will describe a statistical model that cares about who is taking the shot, but does not treat each player as a separate independent case. More on this later…

Data: Arsenal’s Invincibles (Courtesy of StatsBomb)

Once upon a time (in the 2003-04 season), Arsenal FC were brilliant. That squad is still referred to as the Invincibles after finishing the season without a defeat in the league, scoring the most goals and conceding the fewest. Their top scorer, Thierry Henry, finished 4th in the Ballon d’Or voting this season (having finished 2nd the season before). Unfortunately José Mourinho arrived at Chelsea the following season and Arsenal haven’t won the league since.

I’m using Arsenal’s unbeaten league season as an example because StatsBomb have kindly made all this data freely available in their R package.

Here are their league goal scorers:

And here’s where the goals were scored from:

The above plot shows that many of these goals were scored, even though the (StatsBomb) xG was relatively low. In fact the mean xG of the shots they scored was 0.33. This isn’t necessarily a problem as we do see improbable goals. Below is Giorgian De Arrascaeta’s contender for the 2020 FIFA Puskas award. Was anyone expecting him to score this chance? Could he do it again?

An ideal xG model would correctly predict every goal without error, but the many sources of variability in the game means that this isn’t happening any time soon. A Bayesian model (such as the one I’m proposing) will include uncertainty in it’s predictions, letting us know when it can narrow down a predicted xG, and when there is a larger range of credible values based on the available information.

Would Henry have scored it?

Another feature that I’ve introduced to the model is the relationship between the data from different players. I want the model to distinguish between whether a team creates an opportunity for their top scorer, or their full-back who has never scored. One is clearly preferable, and should have a higher xG to reflect this. Why would this matter? Shooting from wide positions may (on average) be unlikely to pay off, but if your team has a winger who is especially adept at it, then it may be a strategy they should pursue.

For instance, Giorgian De Arrascaeta may have had a higher chance of scoring that bicycle kick when you consider that he was also nominated for the 2018 FIFA Puskas award for scoring another acrobatic volley.

The practical issue with considering each player separately is you now have many, smaller datasets. Larger datasets contain more information allowing for model parameters to be estimated more precisely. This sometimes encourages us to throw all our data into a single population and pretend we have a larger dataset. Your software will be happy, since it won’t know the difference, but you will lose the valuable player-specific information.

Bayesian models can do even better than this though. Consider some data that was collected from Arsenal’s defensive midfielder, Gilberto Silva. He scored 3 league goals in their invincible season, but his primary duties were defensive. He had different characteristics than Thierry Henry, but there is some commonality to take advantage of here. If Gilberto Silva scores an opportunity that gives me some information about whether Thierry Henry could have scored it too. How much information? That depends on how similar they are. Unless we tell the model, it will assume we cannot learn anything about these players from the other. Both were professional footballers. Neither was a hockey player, or a tree, or a kitten - though a statistical model could not intuit this. If the data did indicate that they were in fact very different players, then the special model structure that we are using would recognise this and not share information between them to the same extent.

If the above concept make sense to you, then congratulations - you appreciate the utility of multi-level (partial-pooling) Bayesian models. This sharing of information is one of many reasons Bayesian methods can perform so well on small (and imperfect) datasets.

What does the Model do?

We have a model that describes uncertainty (using probability) and makes both team-level and player-specific predictions. Here are some examples:

How about Dennis Bergkamp’s dinked finish when clear through on goal against Birmingham. Remember it? Me neither - here’s where the shot was taken from:

And our predicted xG is shown below, both for Dennis Bergkamp (upper) and for the case where any Arsenal player could be shooting (lower). Here is a great example of being able to make a better prediction conditional on the information of who is taking the shot. The model has identified that Bergkamp was very capable of scoring these kind of chances and was therefore able to identify a narrow range of very high xG values. However, if we were considering a generic player in the Arsenal team, there is more uncertainty in our prediction.

What about Thierry Henry’s long range goal against Man Utd? (Note that the straight arrow in the below plot does not reflect the true trajectory of his shot).

OK, so I wouldn’t have seen that one coming either ….but I would have given it more of a chance knowing who was shooting.

Here is a final example - a shot from Gilberto Silva, on his stronger foot, which was saved by Neil Sullivan (who I’d completely forgotten had signed for Chelsea that season). I thought this was worth looking at because StatsBomb’s xG suggests this was a very good chance.

Our model did not expect him to score, and also predicted that Freddie Ljungberg would have missed. Henry (unsurprisingly) is expected to have had a better chance, but interestingly, our model thinks that Arsenal’s goalscoring winger Robert Pirès would have been most likely to score this opportunity.

Final Thought: Making Sense of Probabilistic Predictions

What should we make of the above predictions? The single values (point estimates) provided by analytics companies may be a bit easier to read, but I’m suggesting that they are not as useful. We should want our models to tell us when they are not sure. There is more information in a probabilistic prediction than a point estimate, which means you can go from the former to the latter, but not vice-versa. The type of model we have discussed in this article has the added benefit of sharing information between different players in a mathematically coherent way (see part 2) for the technical details).

Bayes

Introduction

Firstly, Bayesian…

• Statistics
• Inference
• Modelling
• Updating
• Data Analysis

…can boradly be considered the same thing (certainly for the purposes of this post): the application of Bayes theorem to quantify uncertainty.

Depending on your background, you may have preconceptions about Bayesian methods being specialist or complicated. However, analysts and researchers in many domains are increasingly experimenting with the growing catalogue of open source software and resources that is making probabilistic programming more accessible.

What does a Bayesian approach provide?

Bayesian statistics is not the only way to account for uncertainty in calculations. The below points describe what a Bayesian approach offers, that others don’t. Note that I am only really discussing methods involving probability here, though alternative approaches are available.

Challenges & Difficulties

So why would anyone ever not use Bayesian models when making predictions?

Conclusions

Regardless of whether you believe we exist in a deterministic universe or not, you will never have perfect state of knowledge describing your problem: uncertainty exists, so we need a sensible and safe way of accounting for it. It’s difficult to escape the implications of the Maths - for instance, in avoiding quantifying uncertainty, we are often making some implicit assumptions (the implications of which can be difficult to justify) that we may not want to propagate into our decision-making. There is generally a trade-off for apparent conveniences.

I believe that Bayesian statistics is actually well suited to traditional engineering problems, which are concerned with managing risk when confronted with small, messy datasets and models with plenty of uncertainty. As suggested in the earlier description of confidence intervals, frequentist statistics defines probability based on occurrences of events following a large number of trials or samples. When trying to understand the behaviour of small datasets (or even unique structural systems), Bayesian statistics can shine by comparison.

Very large datasets may contain enough information to precisely estimate parameters in a model using conventional machine learning methods, and so it becomes less worthwhile running simulations to characterise variability. But how common are these big data problems in science and engineering? Sometimes large populations of data are better described as multiple smaller constituent groups, after accounting for key differences between them. Bayesian statistics has a very useful way of managing such problems by structuring models hierarchically. This method allows for partial pooling of information between groups, so that predictions account for the variability and commonality between groups. I will provide a detailed example of this in a future post.

Bayesian inference requires (computational and personal) effort to apply. But it provides results that are generally more interpretable and closely related to the actual questions we want to answer. Whether or not these methods are worth learning will of course depend on personal circumstances. I encountered this field during my PhD, and so had plenty of time to read and play with them, which I appreicate is a privelidged position to be in.

Introductions

So there are a couple of key topics discussed here: logistic regression, and Bayesian inference. Before jumping straight into the example application, I’ve provided some very brief introductions below.

from sklearn.linear_model import LogisticRegression

model = LogisticRegression()
model.fit(X = dataset['input_variables'], y = dataset['predictions'])

model_fit <- glm(formula = preditions ~ input_variables,
                 data = dataset, family = binomial(link = 'logit'))

Example Application: Probability of Detection

To demonstrate how a Bayesian logistic regression model can be fit (and utilised), I’ve included an example from one of my papers. Engineers make use of data from inspections to understand the condition of structures. Modern inspection methods, whether remote, autonomous or manual application of sensor technologies, are very good. They are generally evaluated in terms of the accuracy and reliability with which they size damage. Engineers never receive perfect information from an inspection, such as:

• There is a crack of exact length 30 mm and exact depth 5 mm at this exact location, or
• There is definitely no damage at this location.

For various reasons, the information we receive from inspections is imperfect and this is something that engineers need to deal with. As a result, providers of inspection services should be requested to provide some measure of how good their product is. This typically includes some measure of how accurately damage is sized and how reliable an outcome (detection or no detection) is.

This example will consider trials of an inspection tool looking for damage of varying size, to fit a model that will predict the probability of detection for any size of damage. Since various forms of damage can initiate in structures, each requiring inspection methods that are suitable, let’s avoid ambiguity and imagine we are only looking for cracks.

set.seed(1008)

N <- 30; lower <- 0; upper <- 10; alpha_true <- -1; beta_true <- 1

depth <- runif(n = N, min = lower, max = upper)

PoD_1D <- function(depth, alpha_1D, beta_1D){
  PoD <- exp(alpha_1D + beta_1D * log(depth)) / (1 + exp(alpha_1D + beta_1D * log(depth)))
  return (PoD)
}

pod_df <- tibble(depth = depth, det = double(length = N))

for (i in seq(from = 1, to = nrow(pod_df), by = 1)) {
  
  pod_df$det[i] = rbernoulli(n = 1, 
                             p = PoD_1D(depth = pod_df$depth[i], 
                                       alpha_1D = alpha_true, 
                                       beta_1D = beta_true))
  
}

PoD_model <- cmdstanr::cmdstan_model(stan_file = 'PoD_model.stan')

Warning in readLines(stan_file): incomplete final line found on
'PoD_model.stan'

PoD_model$print()

data {

  int <lower = 0> N; // Defining the number of defects in the test dataset
  array [N] int <lower = 0, upper = 1> det; // A variable that describes whether each defect was detected (1) or not (0)
  vector <lower = 0> [N] depth; // A variable that describes the corresponding depth of each defect
  
  int <lower = 0> K; // Defining the number of probabilistic predictions required from the model
  vector <lower = 0> [K] depth_pred;
  
}

parameters {
  
  // The (unobserved) model parameters that we want to recover
  real alpha;
  real beta;
  
}

model {

  // A logistic regression model relating the defect depth to whether it will be detected
  det ~ bernoulli_logit(alpha + beta * log(depth));
  
  // Prior models for the unobserved parameters
  alpha ~ normal(0, 1);
  beta ~ normal(1, 1);

}

generated quantities {
  
  // Using the fitted model for probabilistic prediction.
  // K posterior predictive distributions will be estimated for a corresponding crack depth
  vector [K] postpred_pr;
  
  for (k in 1:K) {
    
    postpred_pr[k] = inv_logit(alpha + beta * log(depth_pred[k]));
    
  }
  
}

K <- 50; depth_pred <- seq(from = lower, to = upper, length.out = K)

PoD_fit <- PoD_model$sample(data = list(N = N, det = pod_df$det, depth = pod_df$depth,
                                        K = K, depth_pred = depth_pred), seed = 2408)

PoD_samples <- PoD_fit |> DomDF::tidy_mcmc_draws()

head(x = PoD_samples, n = 5)

# A tibble: 5 × 4
  Parameter Chain Iteration value
  <chr>     <int>     <int> <dbl>
1 lp__          1         1 -15.6
2 lp__          1         2 -15.2
3 lp__          1         3 -15.5
4 lp__          1         4 -16.0
5 lp__          1         5 -16.2