Sage 100

 View Only
Expand all | Collapse all

Hidden characters appearing in Sage 100 after copying data from Excel

  • 1.  Hidden characters appearing in Sage 100 after copying data from Excel

    Posted 12-07-2025 16:05

    Hello everyone,

    I'm hoping someone has encountered and successfully resolved this exact issue. We are running into a major problem after copying Inventory Item Codes (CI_Item) from Excel into Sage 100.

    The Problem

    • Duplicate Records: The Item Lookup and Data File Display and Maintenance (DFDM) show the same item code twice: once with a clean code and once with a code containing invisible/non-printable characters (introduced via the copy/paste from Excel).

    • Data Integrity Risk: If we try to edit the "bad" item in DFDM, it deletes all associated history, which is unacceptable.

    • Solution Found (Manual): We discovered that using the Delete and Change Items Utility successfully merges the bad item into the good item, preserving the history.

    The Challenge

    We have approximately 300 items to fix, and we need a bulk solution.

    • Export Failure: When I export the list of "bad" item codes from Sage, the export file (CSV/TXT) does not retain the hidden characters.

    • Utility Import Failure: When I import this "cleaned" list back into the Delete and Change Items Utility, the utility thinks the "bad" item code is identical to the "good" item code and therefore doesn't perform the merge/change operation.

    The utility only works if I manually select the bad item code from the lookup list, which is too time-consuming for 300 records.

    🙏 Questions for the Community

    1. Has anyone found a way to export the Item Code list while preserving the non-printable characters (like CR or LF) so the utility can process them in bulk?

    2. Is there a known third-party utility or script that can clean these specific duplicate keys in the CI_Item file?

    Any guidance on how to get the hidden characters into an importable file format would be greatly appreciated!

    Hidden Characters


    ------------------------------
    Russell Berman
    SAGE 100 Reseller
    Business Solutions Providers
    Hackensack
    ------------------------------


  • 2.  RE: Hidden characters appearing in Sage 100 after copying data from Excel

    Posted 12-07-2025 16:20

    Have you tried using Crystal to export the data?  I would try using tab-delimited format as the file type.



    ------------------------------
    Doug Higgs
    Midwest Commerce Solutions, Inc
    (312) 315-0960
    Chauffeur, Chef, and Personal Assistant to Sprinkles
    ------------------------------



  • 3.  RE: Hidden characters appearing in Sage 100 after copying data from Excel

    Posted 12-07-2025 16:33

    Yes, it exports a space in place of the bad characters. I tried VI export also. I believe I need to get this character exported ß

    Item


    ------------------------------
    Russell Berman
    SAGE 100 Reseller
    Business Solutions Providers
    Hackensack
    ------------------------------



  • 4.  RE: Hidden characters appearing in Sage 100 after copying data from Excel

    Posted 12-07-2025 17:00

    Can you export to Unicode format?https://www.tek-tips.com/threads/crystal-9-unicode-and-special-characters.706950/



    ------------------------------
    Doug Higgs
    Midwest Commerce Solutions, Inc
    (312) 315-0960
    Chauffeur, Chef, and Personal Assistant to Sprinkles
    ------------------------------



  • 5.  RE: Hidden characters appearing in Sage 100 after copying data from Excel

    Posted 12-07-2025 19:42

    You may have to install a Unicode font. It may be worth trying.  Per AI:

    Popular and recommended Unicode fonts include:

    • Arial Unicode MS: This font is widely used for multi-language support and was often included with Microsoft Office installations.
    • Segoe UI Symbol: Useful for displaying various symbols and icons.
    • Other options: Fonts like MS Gothic (for Japanese characters) or general Unicode fonts such as Gentium or GNU FreeFont are also available, depending on specific language requirements. 
    Installation Process (Windows):
    1. Download the font file (usually .ttf or .otf format).
    2. Right-click the font file and select Install, or copy the file into the C:\Windows\Fonts folder. 
    Step 2: Configure the Crystal Report
    Once the font is installed, follow these steps in the Crystal Reports designer:
    1. Open the report in Crystal Reports designer.
    2. Right-click the text object, database field, or formula field that needs to display Unicode characters.
    3. Select Format Field (or Format Object).
    4. Navigate to the Font tab in the Format Editor.
    5. In the Font list, select the installed Unicode font (e.g., Arial Unicode MS or Segoe UI Symbol).
    6. Click OK to apply the changes. 



    ------------------------------
    Doug Higgs
    Midwest Commerce Solutions, Inc
    (312) 315-0960
    Chauffeur, Chef, and Personal Assistant to Sprinkles
    ------------------------------



  • 6.  RE: Hidden characters appearing in Sage 100 after copying data from Excel

    Posted 12-07-2025 20:07

    I added the β to the end of an inventory item and it exported correctly using the TTX format.  It worked with the Segoe UI Symbol font and the Arial font.



    ------------------------------
    Doug Higgs
    Midwest Commerce Solutions, Inc
    (312) 315-0960
    Chauffeur, Chef, and Personal Assistant to Sprinkles
    ------------------------------



  • 7.  RE: Hidden characters appearing in Sage 100 after copying data from Excel

    Posted 12-07-2025 20:52

    Thank you Doug. I really appreciate you helping with this. I tried doing this and it exports a blank space. I wonder if it's not actually the ß character and it's actually a hidden command that shows like that. 

    Item

    n formatting?



    ------------------------------
    Russell Berman
    SAGE 100 Reseller
    Business Solutions Providers
    Hackensack
    ------------------------------



  • 8.  RE: Hidden characters appearing in Sage 100 after copying data from Excel

    Posted 12-07-2025 21:11

    What font are you using in Crystal reports for the item code field?



    ------------------------------
    Doug Higgs
    Midwest Commerce Solutions, Inc
    (312) 315-0960
    Chauffeur, Chef, and Personal Assistant to Sprinkles
    ------------------------------



  • 9.  RE: Hidden characters appearing in Sage 100 after copying data from Excel

    Posted 12-07-2025 21:13

    ...and if you preview the report i Crystal do you see the symbol(s)?



    ------------------------------
    Doug Higgs
    Midwest Commerce Solutions, Inc
    (312) 315-0960
    Chauffeur, Chef, and Personal Assistant to Sprinkles
    ------------------------------



  • 10.  RE: Hidden characters appearing in Sage 100 after copying data from Excel

    Posted 12-07-2025 21:54

    I don't see the symbol in the crystal report.



    ------------------------------
    Russell Berman
    SAGE 100 Reseller
    Business Solutions Providers
    Hackensack
    ------------------------------



  • 11.  RE: Hidden characters appearing in Sage 100 after copying data from Excel

    Posted 12-07-2025 22:10

    That is odd. I inserted the symbol in Word (Insert, Symbol), then copied and pasted it to the end of an item number by editing in dfdm.  I was able to preview and export and see the symbol.



    ------------------------------
    Doug Higgs
    Midwest Commerce Solutions, Inc
    (312) 315-0960
    Chauffeur, Chef, and Personal Assistant to Sprinkles
    ------------------------------



  • 12.  RE: Hidden characters appearing in Sage 100 after copying data from Excel

    Posted 12-07-2025 22:40

    Doug, I don't think Sage will allow you to import that character back, which I believe is what Russel is trying to do. I might be wrong. I did this a few years back.



    ------------------------------
    Kenny Daniel
    TechnoClarity, Inc.
    ------------------------------



  • 13.  RE: Hidden characters appearing in Sage 100 after copying data from Excel

    Posted 12-07-2025 22:57

    I dont think its actually that symbol. I think its a space command in excel but shows up as that symbol. 



    ------------------------------
    Russell Berman
    SAGE 100 Reseller
    Business Solutions Providers
    Hackensack
    ------------------------------



  • 14.  RE: Hidden characters appearing in Sage 100 after copying data from Excel

    Posted 12-07-2025 21:53

    I tried the 2 you recommended and I tried a few others.



    ------------------------------
    Russell Berman
    SAGE 100 Reseller
    Business Solutions Providers
    Hackensack
    ------------------------------



  • 15.  RE: Hidden characters appearing in Sage 100 after copying data from Excel

    Posted 12-07-2025 21:50

    I have run into this a few years ago. Sage100 will not let you import that character regardless of the font. It's not a valid character as far as for Sage100 is concerned. I only had like 10 items, so I manually renumbered via DFDM, I removed the weird characters and added an A to the item number, both in Item Master and in the History files. Then I was able to Merge the two. I know you have a lot more items. I wish i had a better answer. 



    ------------------------------
    Kenny Daniel
    TechnoClarity, Inc.
    ------------------------------



  • 16.  RE: Hidden characters appearing in Sage 100 after copying data from Excel

    Posted 12-07-2025 21:57

    Thanks Kenny that helps. I dont want to chase my tail on this. Might be a good idea to add the A then merge. Maybe I'll just do it in smaller blocks of items. 



    ------------------------------
    Russell Berman
    SAGE 100 Reseller
    Business Solutions Providers
    Hackensack
    ------------------------------



  • 17.  RE: Hidden characters appearing in Sage 100 after copying data from Excel

    Posted 12-07-2025 22:46

    I'm willing to bet they have Scanco Multi Bin or the old WO module. if memory serves me right i traced it to one of these two.  



    ------------------------------
    Kenny Daniel
    TechnoClarity, Inc.
    ------------------------------



  • 18.  RE: Hidden characters appearing in Sage 100 after copying data from Excel

    Posted 12-07-2025 23:01

    No, they were coping and pasting the whole cell in excel. If there's an enter in the cell it will copy to Sage as that hidden character. They should've double clicked the cell and copied just the data. Not the whole cell.



    ------------------------------
    Russell Berman
    SAGE 100 Reseller
    Business Solutions Providers
    Hackensack
    ------------------------------



  • 19.  RE: Hidden characters appearing in Sage 100 after copying data from Excel

    Posted 12-08-2025 10:50

    If they were caused by copy/paste from Excel it should be CRLF.  Try a Crystal Report with a formula to replace chr(13) and chr(10) with temp characters (like "^", "%"...) then in your VI import to bring them back, do a formula to bring back the original characters.

    If that doesn't work (meaning the invisible characters are stripped by ODBC) try hard coding in the extra characters within the VI job... not relying on having them in the source file.



    ------------------------------
    Kevin Moyes
    Technical Systems Analyst
    Munjal White Consulting Co.
    Toronto ON
    ------------------------------



  • 20.  RE: Hidden characters appearing in Sage 100 after copying data from Excel

    Posted 12-08-2025 11:34

    Last time I addressed this for a client was in August, and we had to go through DFDM to remove lots of bad ß characters from the applicable records.  I did consider a VI import but can't recall how far I looked into it, so I'd be curious to know if anyone gets it working.  This data corruption is often the result of copying/pasting into Sage 100, and not backspacing over any extra spaces/characters.   



    ------------------------------
    Brett Zimmerman
    Net at Work
    Greater Boston Area
    ------------------------------



  • 21.  RE: Hidden characters appearing in Sage 100 after copying data from Excel

    Posted 12-08-2025 12:00

    For what it is worth, we have been able to fix this in Sage 100 Premium with Scripts, but you have to check a zillion tables where that Item code might exist, including Sales Order, SO History, AR Invoice History Detail, etc. etc. etc.  



    ------------------------------
    David Overholt
    Programmer/Project Manager
    DWD Technology Group
    Fort Wayne IN
    (260) 399-8655
    ------------------------------



  • 22.  RE: Hidden characters appearing in Sage 100 after copying data from Excel

    Posted 12-08-2025 12:15

    Are you able to share the script?



    ------------------------------
    Russell Berman
    SAGE 100 Reseller
    Business Solutions Providers
    Hackensack
    ------------------------------



  • 23.  RE: Hidden characters appearing in Sage 100 after copying data from Excel

    Posted 12-08-2025 13:53
      |   view attached

    The attached SQL script is for Sage 100 Premium and comes with all the usual disclaimers - make backups, run in a test company first,  Have the client check the results in test company, etc.

    The script looks for the following characters in the Item Code :

    • Space as first Character
    • Space as the last Character
    • CHAR(10)  (LineFeed)
    • CHAR(9) (Tab)
    • CHAR(13) (Carriage Return)

    We have used this successfully for one of our clients and they reach out to us every 6 months or so to have us re-run it since employees continue to cut/paste from websites and excel and word.

    The list of files in the script has been built over several iterations and corrects over 120 files. Is that EVERY file??  We feel that it is pretty comprehensive but probably not all-encompassing depending on what modules or third-party products are installed. For some files, there are already duplicate records with a "good" item code and a "bad" item code, so script deletes the "bad" record. Other tables where the item code is not part of the key field, it simply cleans the bad characters from the item code field and saves the record. With that said, I would recommend rebuild sort files in all modules after using this. 

    I hope it is helpful at least as a reference. 

     



    ------------------------------
    David Overholt
    Programmer/Project Manager
    DWD Technology Group
    Fort Wayne IN
    (260) 399-8655
    ------------------------------



  • 24.  RE: Hidden characters appearing in Sage 100 after copying data from Excel

    Posted 12-08-2025 14:51

    Thank you so much David. Appreciate that. I'll let you know how it goes.



    ------------------------------
    Russell Berman
    SAGE 100 Reseller
    Business Solutions Providers
    Hackensack
    ------------------------------



  • 25.  RE: Hidden characters appearing in Sage 100 after copying data from Excel

    Posted 12-08-2025 12:13

    Did the history stay there for you? For us when I deleted the bad characters on 1 item it asked to remove the duplicate key. Which removed the original item, and we lost the history. I found if we did the Delete and change utility, it retained the history. 



    ------------------------------
    Russell Berman
    SAGE 100 Reseller
    Business Solutions Providers
    Hackensack
    ------------------------------



  • 26.  RE: Hidden characters appearing in Sage 100 after copying data from Excel

    Posted 12-08-2025 12:07

    See the picture below:

    Line 1, our programmer imported the missing character. 
    Line 2 I selected manually. 
    Delete and Change log shows the first one was skipped because item was not on file.
    The second manual one worked.
    Looking at DFDM the lines look identical.

    The ß character seems to not really be that character in Sage. But it shows like it is.

    Bad Characters



    ------------------------------
    Russell Berman
    SAGE 100 Reseller
    Business Solutions Providers
    Hackensack
    ------------------------------



  • 27.  RE: Hidden characters appearing in Sage 100 after copying data from Excel

    Posted 12-08-2025 12:36

    It is absolutely not the ß character.  That is just how invisible characters are shown in DFDM.



    ------------------------------
    Kevin Moyes
    Technical Systems Analyst
    Munjal White Consulting Co.
    Toronto ON
    ------------------------------



  • 28.  RE: Hidden characters appearing in Sage 100 after copying data from Excel

    Posted 12-08-2025 12:57

    Yes, That's what we're noticing. We're trying to figure out what character it actually is so we can import it.

    Bad Items



    ------------------------------
    Russell Berman
    SAGE 100 Reseller
    Business Solutions Providers
    Hackensack
    ------------------------------



  • 29.  RE: Hidden characters appearing in Sage 100 after copying data from Excel

    Posted 12-08-2025 13:25

    For just one I'd use the Sage interface / lookup when doing the renumber.  For more than one, try this as a button script:

    retVal = 0
    sItemCode = "" : sOutputText = ""
    retVal = oBusObj.GetValue("ItemCode",sItemCode)
    sOutputText = sOutputText & "Item: " & sItemCode & vbCrlf
    if len(sItemCode) > 0 then
        for i = 1 to len(sItemCode)
            sOutputText = sOutputText & "Ascii(" & cStr(i) & "): " & cStr(Asc(mid(sItemCode,i,1))) & vbCrlf
            sOutputText = sOutputText & "Unicode(" & cStr(i) & "): " & cStr(AscW(mid(sItemCode,i,1))) & vbCrlf
        next ' i
    end if
    retVal = oSession.AsObject(oSession.UI).MessageBox("", sOutputText)

    ...it should give you a list of the ascii characters that make up the Item Code.



    ------------------------------
    Kevin Moyes
    Technical Systems Analyst
    Munjal White Consulting Co.
    Toronto ON
    ------------------------------



  • 30.  RE: Hidden characters appearing in Sage 100 after copying data from Excel

    Posted 12-08-2025 13:33

    Thanks a lot Kevin. We'll give this a shot.



    ------------------------------
    Russell Berman
    SAGE 100 Reseller
    Business Solutions Providers
    Hackensack
    ------------------------------



  • 31.  RE: Hidden characters appearing in Sage 100 after copying data from Excel

    Posted 12-08-2025 13:28

    Can your program get the length of the field?  If so, you could have a program examine each character until it discovers a character that is not in A -Z or 0 -9.  When it discovers the strange character then add "ZZZ" to the end of the last valid letter or number.  For example:

    A$=ItemCode$

    Dim string NewItem$

    read (len( A$)); 'returns 10

    For X = 1 to 10 read(A$(X) until A$(X,1) not in [A,B,C, ... 0,1,2,3,etc]

    Let NewItem$= A$(X-1)+"ZZZ"

    Write(NewItem$)

    Delete A$



    ------------------------------
    Doug Higgs
    Midwest Commerce Solutions, Inc
    (312) 315-0960
    Chauffeur, Chef, and Personal Assistant to Sprinkles
    ------------------------------



  • 32.  RE: Hidden characters appearing in Sage 100 after copying data from Excel

    Posted 12-08-2025 13:37

    Could possibly work. Thank you for this. I'll send it over the programmer and report back.



    ------------------------------
    Russell Berman
    SAGE 100 Reseller
    Business Solutions Providers
    Hackensack
    ------------------------------



  • 33.  RE: Hidden characters appearing in Sage 100 after copying data from Excel

    Posted 12-08-2025 22:00

    Cleanup of Item Codes Containing Hidden Characters in Sage 100 Using Delete & Change and Automated CI_Item Reconciliation


    Overview

    During review of CI_Item, several item codes were found to contain trailing hidden characters caused by past imports or copy-and-paste activity. These corrupted values resulted in duplicate items, lookup problems, and inconsistent item maintenance behavior.

    Complicating the issue:

    • Some items had both a clean and a bad version,

    • While others existed only in their corrupted (hidden-character) form.

    To ensure all items could be merged properly, we first imported missing clean versions before performing the cleanup.


    Symptoms

    • Duplicate item codes (clean vs. corrupted)

    • Some corrupted items with no corresponding clean version

    • Hidden characters at the end of the item code

    • Lookup/search inconsistencies

    • Difficulty selecting the correct item in screens and reports


    Cleanup Procedure

    1. Import Missing Clean Item Codes into CI_Item

    For any item where only the corrupted version existed, we imported a clean item code into CI_Item.

    This ensured that every corrupted item had a valid clean target for the Delete & Change process.


    2. Import Clean Item Codes Into Delete & Change (Both From & To)

    We then imported the clean item code into:

    • From Item Code

    • To Item Code

    This created a full placeholder mapping for every item requiring cleanup:

    FromItem = CLEAN123 ToItem = CLEAN123

    3. Automatically Replace "From" With Actual Bad Item Codes

    A custom program scanned CI_Item for any item codes that had an extended length as compared with the IM_DeleteChangeItems table.

    For each match, the program:

    • Identified the corrupted version

    • Updated the Delete & Change "From" field by replacing the clean value with the actual stored bad item code

    Resulting in mappings like:

    FromItem = CLEAN123<hidden_char> ToItem = CLEAN123

    This allowed Delete & Change to correctly merge each bad item into its clean version.


    4. Run the Delete & Change Utility

    After all mappings were corrected, the Delete & Change utility was executed to:

    • Consolidate corrupted item codes into the clean item codes

    • Remove duplicate entries

    • Update all references and internal links to the correct item code


    5. Verification

    Post-processing validation confirmed:

    • All hidden-character item codes were removed

    • All items now exist under a single, clean item code

    • Lookups and searches work consistently

    • No unpaired corrupted item codes remain


    Outcome

    The item master file was fully normalized.
    Any item that previously existed only in a corrupted state was paired with an imported clean version and correctly merged.
    All variations were consolidated into proper clean item codes.



    ------------------------------
    Russell Berman
    SAGE 100 Reseller
    Business Solutions Providers
    Hackensack
    ------------------------------