Sage 100

 View Only
Expand all | Collapse all

Any way to control Excessive V/I Log Files since v2019

  • 1.  Any way to control Excessive V/I Log Files since v2019

    Posted 05-29-2020 20:52
    Edited by Alnoor Cassim 05-30-2020 13:32
    Simple question at the bottom.

    Very large customer on Advanced was just upgraded from 2014 to 2019 and they have a ton of V/I export and imports they run daily. It has been that way for a long time and it's been running ok for years. A new feature since v2019 is to create log files in \Mas_System\LogFiles folder for various events.

    * DFDM record changes are logged
    * Role Maintenance changes are logged
    * Some other events logged but I don't know have the list.
    * V/I export and import jobs, the Job Log is no longer written to \VI folder as VIW?NN.M4T (where ? = X for export or I for import)

    The complaint to Sage was it only stores log info from last job not previous ones. So we saw in the 2019 Sage demo at MotM they moved this to LogFiles folder as: 
    VI_VIW?NN_YYYYMMDD_hhmmss.M4T   (where YYYYMMDD is the date and hhmmss is the time)

    So every single export or import has a separate discrete physical file. Seems cool except not for this customer. Their IT emailed that since last night the LogFiles folder grew over 10 GB. I saw it was primarily because some huge exports had individual log files reaching 2 GB file. There were also over 1,000 M4T files. Previous to v2019 remember it's always the same single log file per job that was getting re-used that's why there wasn't as much disk space used and the same number of log files.

    Question: Can this LOGGING feature be turned off for V/I ?

    Looking for a front door way to turn this off. I've done a cursory search thru Help and Sage KB but no joy finding info on this topic. I could resort to changing the programming but prefer a front door.

    ------------------------------
    Alnoor Cassim

    Accounting Systems, Inc. (ASI)
    Email: alnoor@asifocus.com
    Orange County, CA
    ------------------------------


  • 2.  RE: Any way to control Excessive V/I Log Files since v2019

    Posted 05-30-2020 04:45
    Haven't ran into that yet but rather than change sage programming, wouldn't it be easier to set up a .bat or .vbs executed by Windows Task Scheduler to clean up the log files that are x days old?

    ------------------------------
    David Speck II
    Tennessee Software Solutions
    ------------------------------



  • 3.  RE: Any way to control Excessive V/I Log Files since v2019

    Posted 05-30-2020 13:12
    David I had brought that up already and it would be my normal Plan B but there are other factors:

    a. Even with clean up of older logs thru BAT or VBS, the added disk space makes their backup take longer to complete. It affects people in the early morning shift who say the system is dragging (b/c backup is still running). 
    b. Customer does not like how the location of log files cannot be user defined.
    c. "The Log Table is empty" msg appears when you click Job Log inside Export Job Maintenance despite the files being there in LogFiles folder. This is important for them. AFAIK, there is still not a report or other substitute to viewing the log now other than DFDM. The logs are still not exportable (except for my custom solution and 2020 release in Sept), and still not dictionary defined.

    Overall, customer is very peeved and views all of this very negatively, as a big takeaway for them.
    They say "why do have to put in workarounds for dealing with Sage caused issues" in terms of changing backup schedules and adding BAT / VBS files to delete files, and for Sage not providing a front-end way to turn off or revert the feature.

    We do custom programming for them anyway so I may revert the behavior to the old way or let them choose on a per export/import job basis or change the logging location, but I wanted to verify there wasn't a front-end way to deal with this already.

    I personally think the LogFiles concept as far as security events go is a cool Sage feature but it's not fully fleshed out for V/I. 


    ------------------------------
    Alnoor Cassim

    Accounting Systems, Inc. (ASI)
    Email: alnoor@asifocus.com
    Orange County, CA
    ------------------------------



  • 4.  RE: Any way to control Excessive V/I Log Files since v2019

    Posted 05-30-2020 13:40
    That makes sense. When you put it that way, it certainly feels like a half baked feature.





    Thanks

    David Speck

    TSS

    Sage | Sage 100 Certified Consultant







  • 5.  RE: Any way to control Excessive V/I Log Files since v2019

    Posted 05-30-2020 13:43
    It would be nice if the log files were saved as a pdf in the paperless office style. It seems lazy to create a SOW and "enhance" the log files, yet still write the data to a proprietary database that can only be viewed using DFDM or odbc. No options to delete log file history, and no utility to view the log files.

    ------------------------------
    Doug Higgs
    Midwest Commerce Solutions, Inc
    (312) 315-0960
    Assistant to the Traveling Secretary
    ------------------------------



  • 6.  RE: Any way to control Excessive V/I Log Files since v2019

    Posted 05-31-2020 17:09
    Isn't the VI Log Files being changed in the PU for 2020? Now you will see a drill down for it in the Event viewer and you can export it if you would like to. I am guessing that this doesn't change the way they still have a file created for each import. In that case can you not do perform logic to clean up or move the files from previous imports each time it is run? What would really be interesting is if you can use MKLink to do a Symbolic Link so it moves it to another location but the system still thinks it is local. 

    so I think with the new change to log files and maybe if the symbolic Link works, then the customer can get all they want. MAYBE.Who knows Customers can be funny that way.

    ------------------------------
    Todd Martin - President
    MBA Business Software (https://www.mbabsi.com)
    ------------------------------



  • 7.  RE: Any way to control Excessive V/I Log Files since v2019

    Posted 05-31-2020 22:31
    Edited by Alnoor Cassim 12-18-2020 13:25

    Todd you nailed it! Activity Log drill-down is the front-end for viewing all the export and import job logs!! When I saw the 90 Minds Sage Dev Roundatable recently, I saw that and confused it to mean it was coming in 2020 PU1, not realizing it was already in 2019. Works nice after filtering for V/I module. Interestingly, the last job log for imports is viewable directly in Import Job Maintenance using the Job Log button but in Export Job Maintenance "The Log Table is empty" appears (bug).

    I had considered Perform Logic but they have around 50 exports and 20 imports so it seemed too much to manually add to each job. I could instead merge the Perform records into the Perform tables or just override VI_Export_UI and VI_Import_UI. I'll give them some options. I've never used mklink to do a symbolic link. I'll look into it!

    On the last 90 Minds Sage Dev roundtable, Fred M talked about V/I enhancements coming for 2020 PU1 in Sept. 

    • For import jobs, on Job Log display click new button to export failed records to TXT file saved to source file location folder (but not also emailed). Limited to imports of Delimited and ASCII type.
      (Good stuff but why a button instead of auto-run as that means it cannot run on auto-imports? Why hard-code location to source folder b/c that means you limited yourself to Delimited and ASCII? Why not email?  Quick wins are great but I think this one should be full-baked. I had to create a custom soln to do all that.)
    •  Job Log display to allow filtering
    •  Job Log display to include original row / line from source file
    • 3 additional V/I requests (unstated) from the Sweet 16 list.

    Thank you Todd!

    ------------------------------
    Alnoor Cassim

    Accounting Systems, Inc. (ASI)
    Email: alnoor@asifocus.com
    Orange County, CA
    ------------------------------



  • 8.  RE: Any way to control Excessive V/I Log Files since v2019

    Posted 06-02-2020 09:37
    Elliott is the one who improved the Activity Log so if you want changes, he's the one to ask.
    Writing a VBScript to scan the folder and delete all but the most recent M4T for an import (then scheduling it... every 30 minutes) shouldn't be too hard.

    ------------------------------
    Kevin Moyes
    Technical Systems Analyst
    Munjal White Consulting Co.
    Toronto ON
    ------------------------------



  • 9.  RE: Any way to control Excessive V/I Log Files since v2019

    Posted 06-01-2020 12:11
    This is increasingly important as the small Sage 100 users depart due to never-ending price hikes. The remaining users are larger companies who do have these types of security concerns which means Sage needs to pay attention to these requests or risk losing portions of their remaining installed base.

    And, please, for updates don't just throw a log file into a path where every Sage user is required to have full rights. At some point, an auditor is going to know the questions to ask and will inquire about who can access that folder and how likely it is that the log file could be erased, replaced, or modified.

    I'm already seeing these types of questions on a regular basis which makes me think at some point the only realistic versions are Advanced and Premium which should allow for some safeguarding of data by virtue of service accounts/ SQL.

    ------------------------------
    Wayne Schulz - Schulz Consulting - 860-516-8990
    ------------------------------



  • 10.  RE: Any way to control Excessive V/I Log Files since v2019

    Posted 12-18-2020 13:09
    Edited by Brett Zimmerman 12-18-2020 13:10

    Response from Sage:  "After checking with my team I did confirm that there is currently no option to have VI logs omitted from records. I did fins however that logs can be filtered after being created with Sage 100 2020. I agree with you that VI logs can definitely get out of hand when imports are being run every hour, and believe there should be an option to turn logging off."

    - I assume he meant "deleted" and not "filtered".



    ------------------------------
    Brett Zimmerman
    Net at Work
    Greater Boston Area
    ------------------------------



  • 11.  RE: Any way to control Excessive V/I Log Files since v2019

    Posted 12-18-2020 13:26
    Edited by Alnoor Cassim 12-18-2020 13:26

    Thanks for sharing Brett. I'll vote / comment on your entry in the the Ideas site! BTW I think he meant filtering not deleting. I commented on it above in a previous reply.



    ------------------------------
    Alnoor Cassim

    Email: alnoor@asifocus.com
    Ph: 949-689-9887
    Orange County, CA
    ------------------------------



  • 12.  RE: Any way to control Excessive V/I Log Files since v2019

    Posted 12-18-2020 13:46

    Thanks, @Alnoor Cassim, I actually did see where you mentioned that previously above and thought after-the-fact that it may have been related.



    ------------------------------
    Brett Zimmerman
    Net at Work
    Greater Boston Area
    ------------------------------



  • 13.  RE: Any way to control Excessive V/I Log Files since v2019

    Posted 12-23-2020 15:17

    Voted for it and added my comments. Thanks Brett! https://www5.v1ideas.com/TheSageGroupplc/Sage100ERP/Idea/Detail/40956



    ------------------------------
    Alnoor Cassim

    Email: alnoor@asifocus.com
    Ph: 949-689-9887
    Orange County, CA
    ------------------------------



  • 14.  RE: Any way to control Excessive V/I Log Files since v2019

    Posted 07-12-2021 13:03
    Edited by Brett Zimmerman 07-19-2021 14:36
    This was apparently only addressed for the v2021 release, but then (on 7/19) I found out it's also in v2020.2 (see my next post below).  My v2020.1 client where I ran into this issue could really use this.  If anyone (@Alnoor Cassim?) happens to find out if/how this can apply to v2020, please let me know / post here. 
    Options for Managing Log Files
      • A Setup menu has been added to the Visual Integrator module. Visual Integrator Options has been added to the menu.
      • In Visual Integrator Options you can manage the log files that are generated when running import and export jobs and when automatically scheduled jobs are run.
      • You can specify whether to reuse the existing log files when jobs are run or create new ones. You can also select the location where log files are saved.



    ------------------------------
    Brett Zimmerman
    Net at Work
    Greater Boston Area
    ------------------------------



  • 15.  RE: Any way to control Excessive V/I Log Files since v2019

    Posted 07-19-2021 14:43

    I just found out this resolution is also in v2020.2, of which I located the following in the v2020.2 Release Notes:  "Visual Integrator Job Log Report no longer defaults to print Logs for All existing Jobs.  DevID 112745 Knowledgebase ID 109111".  Apparently the changes made for this Res were fairly substantial and so it was decided to roll it into the next v2020 PU (.2 at the time) and v2021.

    I've asked Sage to update the KB article, as currently for Resolution it shows "Sage Engineering is currently reviewing this issue".  And I've requested they update the Ideas site post to include that it was also fixed in v2020.2.

     



    ------------------------------
    Brett Zimmerman
    Net at Work
    Greater Boston Area
    ------------------------------



  • 16.  RE: Any way to control Excessive V/I Log Files since v2019

    Posted 07-19-2021 15:08
    Thanks for reaching out Brett! I wish they had made that clear to begin with. Also the title of the defect is unclear, not exactly indicating that excessive logging and excessive disk space usage are the real symptoms.

    ------------------------------
    Alnoor Cassim

    Email: alnoor@asifocus.com
    Ph: 949-689-9887
    Orange County, CA
    ------------------------------



  • 17.  RE: Any way to control Excessive V/I Log Files since v2019

    Posted 07-19-2021 15:31
    The Ideas site posting has been updated.  They're now working on the KB article, of which I've since made addt'l mention of @Alnoor Cassim's above comment/suggestion. ​

    ------------------------------
    Brett Zimmerman
    Net at Work
    Greater Boston Area
    ------------------------------