@KevinMoyes brings up a good point, so I started looking at the Upgrade Checklist for 2017. In the 2015 to 2017 and 2016 to 2017, it only mentions:
Update customized panels by selecting Custom Office Utilities menu >
Update Customized Panels to Current Level. Review the placement of all
user-defined fields.
Has anyone other than Kevin tried the 'not updating'? After a product update, running conversion, does the same hold true, just run the utilities?