Tuning Model Triggers for Short-Term Forward Works Planning
Last year we had the first opportunity to run Juno Cassandra on several Local Authority Road Networks here in New Zealand. Feedback from clients was generally good. All clients were happy with the strategic outcomes from the models in terms of budget needs and how we reported this. However, most clients were not 100% happy with the assigned treatments – both in terms of the segments that were selected and the treatment type assigned.
A short house-keeping note: if you received this email in your inbox and feel that you already get too much mail, then feel free to un-subscribe. If you are still interested in following this content, I suggest you just add a shortcut to the blog URL and look in from time to time to see if there is anything interesting or useful to you.
This year, we are going back to the drawing board as far as the Domain Model for NZ Local Authority Road Networks is concerned. We removed some of the black box machine learning models and - for most of the distresses - replaced them with more transparent, traditional models such as the S-Curve Models with the two phases (initialisation and then progression).
We are grateful to one of our clients, representing a large city council, who provided us with their team’s field validated works programme for the next year, including a second-year programme that was validated using the Juno Virtual Field Inspection tool. Below, I refer to this combined set as the “Validated Set”.
For the last week, I have been using this Validated Set to do a painstakingly careful critical analysis of the Cassandra outputs. The objective is to refine the trigger rules to give maximal agreement with the validated two-year programme. The process I am following is shown below. I am iteratively going through this process, slowly improving the treatment selection rule set as well as the Multi Criteria Decision Analysis (MCDA) setup:
I start off with by running Juno Cassandra with the current trigger rule set. For this client, the model input set consists of approximately 3,500 model segments. Running the Cassandra model on my chunky desktop over a 20-year period takes less than a minute, so this is the easy part.
Next, I run an R-script that joins the Cassandra output with the Validated Set. This script automatically assigns to each model segment one of three labels:
Both No Treatment (neither Cassandra nor Validated Set has a treatment).
Both With Treatment (Cassandra and Validated Set has a treatment).
Validated Treatment Only (no treatment assigned by Cassandra, but segment is in the Validated Set).
Cassandra Only (no treatment in Validated Set, but Cassandra assigned a treatment).
I am considering that the Validated Set may contain anomalies, inconsistencies, and room for improvement, so I am careful not to regard the field validated set as “ground truth”. Rather, at this stage, I am focusing only on the “Cassandra Only” sub-set and asking whether these treatments are reasonable or not given the multiple objectives defined in Cassandra’s Multi-Criteria Decision Analysis model.
After each iteration, I take the “Cassandra Only” set and start inspecting segments in this set using the JunoViewer Virtual FIT. I am guided here mainly by experience and subjective judgement, but I am following a thought-process along the following lines:
Mentally absorb” surfacing and pavement age and expected remaining life.
Ditto for traffic (ADT and Heavy Vehicle percentages) and road situation in video.
Scan the values for all distresses as well as rutting and roughness and – where available - FWD or MSD information.
Check the road length – is this a “short” segment?
All the above gives me a mental picture of the road situation. Once I have this in mind, I “drive” the road using the video in Juno’s Virtual FIT, all the while asking myself “if I was faced with a very limited budget, would I spend my available funding here or would I rather assign routine maintenance?”
The Juno Virtual FIT (shown above) is an indispensable tool for me in this process. Also hugely helpful is the fact that this client has performed Video Inferencing of distress using the Juno Intelligence toolset for AI-driven distress and roadside furniture identification.
Using this process, I run down the list of segments that Cassandra had placed in the two-year treatment programme and for each segment I decide whether the segment warrants a place on the highly selective shortlist.
When I run into a segment that I think is a borderline candidate, I flag it as such and keep going until I find a segment that I think does not warrant a treatment over the short term. Once I find such a segment, I scan the condition data and model predictions in conjunction with the trigger rules to determine why this segment ended up in the short term works plan.
Next, I design a new rule, or refine the ruleset, to ensure that this segment will not be selected in the next iteration of the process shown above. This is a somewhat slow iterative process but quite rewarding and informative. This week I made much progress in refining the ruleset, but more work is needed to get it to the level we would like it to be. I also stumbled upon a useful way to compare two works plans to assess whether they hit the mark with respect to the stated (multiple) objectives.
In a later post, I will report on some of my findings - when and why anomalies occur, and the rules that were added to circumvent them. This is likely to be a work in progress, but we hope that in the next month we will have a much-improved model for NZ Local Authorities ready to plug into Cassandra.
I will keep you updated on this process and lessons learned along the way in future posts.