Fixed - here's how:
The data was a string UDF (a time actually in format ##:##)
Added an expression to the connector of:
If ((UDF_TIME) = """", 0, TimeValue(UDF_TIME))
Added this expression to the columns in the report, then the pivot table
Changed the format of the field to Time HH:MM
You may not need the 'If' statement, but I added in case there were fields missing this data so that I didn't see the an Excel error.