Cleanup of Item Codes Containing Hidden Characters in Sage 100 Using Delete & Change and Automated CI_Item Reconciliation
Overview
During review of CI_Item, several item codes were found to contain trailing hidden characters caused by past imports or copy-and-paste activity. These corrupted values resulted in duplicate items, lookup problems, and inconsistent item maintenance behavior.
Complicating the issue:
-
Some items had both a clean and a bad version,
-
While others existed only in their corrupted (hidden-character) form.
To ensure all items could be merged properly, we first imported missing clean versions before performing the cleanup.
Symptoms
-
Duplicate item codes (clean vs. corrupted)
-
Some corrupted items with no corresponding clean version
-
Hidden characters at the end of the item code
-
Lookup/search inconsistencies
-
Difficulty selecting the correct item in screens and reports
Cleanup Procedure
1. Import Missing Clean Item Codes into CI_Item
For any item where only the corrupted version existed, we imported a clean item code into CI_Item.
This ensured that every corrupted item had a valid clean target for the Delete & Change process.
2. Import Clean Item Codes Into Delete & Change (Both From & To)
We then imported the clean item code into:
-
From Item Code
-
To Item Code
This created a full placeholder mapping for every item requiring cleanup:
3. Automatically Replace "From" With Actual Bad Item Codes
A custom program scanned CI_Item for any item codes that had an extended length as compared with the IM_DeleteChangeItems table.
For each match, the program:
Resulting in mappings like:
This allowed Delete & Change to correctly merge each bad item into its clean version.
4. Run the Delete & Change Utility
After all mappings were corrected, the Delete & Change utility was executed to:
5. Verification
Post-processing validation confirmed:
-
All hidden-character item codes were removed
-
All items now exist under a single, clean item code
-
Lookups and searches work consistently
-
No unpaired corrupted item codes remain
Outcome
The item master file was fully normalized.
Any item that previously existed only in a corrupted state was paired with an imported clean version and correctly merged.
All variations were consolidated into proper clean item codes.
------------------------------
Russell Berman
SAGE 100 Reseller
Business Solutions Providers
Hackensack
------------------------------
Original Message:
Sent: 12-08-2025 13:37
From: Russell Berman
Subject: Hidden characters appearing in Sage 100 after copying data from Excel
Could possibly work. Thank you for this. I'll send it over the programmer and report back.
------------------------------
Russell Berman
SAGE 100 Reseller
Business Solutions Providers
Hackensack
------------------------------
Original Message:
Sent: 12-08-2025 13:28
From: Doug Higgs
Subject: Hidden characters appearing in Sage 100 after copying data from Excel
Can your program get the length of the field? If so, you could have a program examine each character until it discovers a character that is not in A -Z or 0 -9. When it discovers the strange character then add "ZZZ" to the end of the last valid letter or number. For example:
A$=ItemCode$
Dim string NewItem$
read (len( A$)); 'returns 10
For X = 1 to 10 read(A$(X) until A$(X,1) not in [A,B,C, ... 0,1,2,3,etc]
Let NewItem$= A$(X-1)+"ZZZ"
Write(NewItem$)
Delete A$
------------------------------
Doug Higgs
Midwest Commerce Solutions, Inc
(312) 315-0960
Chauffeur, Chef, and Personal Assistant to Sprinkles
Original Message:
Sent: 12-08-2025 12:56
From: Russell Berman
Subject: Hidden characters appearing in Sage 100 after copying data from Excel
Yes, That's what we're noticing. We're trying to figure out what character it actually is so we can import it.
------------------------------
Russell Berman
SAGE 100 Reseller
Business Solutions Providers
Hackensack
Original Message:
Sent: 12-08-2025 12:36
From: Kevin Moyes
Subject: Hidden characters appearing in Sage 100 after copying data from Excel
It is absolutely not the ß character. That is just how invisible characters are shown in DFDM.
------------------------------
Kevin Moyes
Technical Systems Analyst
Munjal White Consulting Co.
Toronto ON
Original Message:
Sent: 12-08-2025 12:07
From: Russell Berman
Subject: Hidden characters appearing in Sage 100 after copying data from Excel
See the picture below:
Line 1, our programmer imported the missing character.
Line 2 I selected manually.
Delete and Change log shows the first one was skipped because item was not on file.
The second manual one worked.
Looking at DFDM the lines look identical.
The ß character seems to not really be that character in Sage. But it shows like it is.
------------------------------
Russell Berman
SAGE 100 Reseller
Business Solutions Providers
Hackensack
Original Message:
Sent: 12-08-2025 10:50
From: Kevin Moyes
Subject: Hidden characters appearing in Sage 100 after copying data from Excel
If they were caused by copy/paste from Excel it should be CRLF. Try a Crystal Report with a formula to replace chr(13) and chr(10) with temp characters (like "^", "%"...) then in your VI import to bring them back, do a formula to bring back the original characters.
If that doesn't work (meaning the invisible characters are stripped by ODBC) try hard coding in the extra characters within the VI job... not relying on having them in the source file.
------------------------------
Kevin Moyes
Technical Systems Analyst
Munjal White Consulting Co.
Toronto ON
Original Message:
Sent: 12-07-2025 23:00
From: Russell Berman
Subject: Hidden characters appearing in Sage 100 after copying data from Excel
No, they were coping and pasting the whole cell in excel. If there's an enter in the cell it will copy to Sage as that hidden character. They should've double clicked the cell and copied just the data. Not the whole cell.
------------------------------
Russell Berman
SAGE 100 Reseller
Business Solutions Providers
Hackensack
Original Message:
Sent: 12-07-2025 22:46
From: Kenny Daniel
Subject: Hidden characters appearing in Sage 100 after copying data from Excel
I'm willing to bet they have Scanco Multi Bin or the old WO module. if memory serves me right i traced it to one of these two.
------------------------------
Kenny Daniel
TechnoClarity, Inc.
Original Message:
Sent: 12-07-2025 21:56
From: Russell Berman
Subject: Hidden characters appearing in Sage 100 after copying data from Excel
Thanks Kenny that helps. I dont want to chase my tail on this. Might be a good idea to add the A then merge. Maybe I'll just do it in smaller blocks of items.
------------------------------
Russell Berman
SAGE 100 Reseller
Business Solutions Providers
Hackensack
Original Message:
Sent: 12-07-2025 21:50
From: Kenny Daniel
Subject: Hidden characters appearing in Sage 100 after copying data from Excel
I have run into this a few years ago. Sage100 will not let you import that character regardless of the font. It's not a valid character as far as for Sage100 is concerned. I only had like 10 items, so I manually renumbered via DFDM, I removed the weird characters and added an A to the item number, both in Item Master and in the History files. Then I was able to Merge the two. I know you have a lot more items. I wish i had a better answer.
------------------------------
Kenny Daniel
TechnoClarity, Inc.