Engine Limitation Bad Saves

[Talisman]

I have been getting several "Bad Saves" with Beta 4.

When this happens I get this in the error.log

 

[Pieter Boelen]

I think I've seen that one occasionally very many months ago, but I believe it is just one of those random things that we cannot track down at all.
It is unlikely that it is related to specific changes made for Beta 4.

You can try the old tricks of removing the "options" file and emptying out your SAVE folder of everything you don't directly need.

 

[salonikasurf]

This seems to be a problem - the further I get into the game (Beta 4) the more times I get bad saves - quicksave seems to work better.
But still an issue.

 

[Pieter Boelen]

This seems to be a problem - the further I get into the game (Beta 4) the more times I get bad saves - quicksave seems to work better.
But still an issue.

There has been a potential issue with this for years, as far as I'm aware. There is no reason why it should be either better or worse now.
All I can suggest is to delete the file named "options" from your main game folder, don't have too many different profiles at the same time and remove all files from your SAVE folder that you don't need.

 

[Levis]

How much RAM do you have?

 

[Talisman]

How much RAM do you have?

I have 16 Gb of RAM
Windows 8.1 OS

 

[Talisman]

There has been a potential issue with this for years, as far as I'm aware. There is no reason why it should be either better or worse now.
All I can suggest is to delete the file named "options" from your main game folder, don't have too many different profiles at the same time and remove all files from your SAVE folder that you don't need.

Beta 4 is the first time I have regularly experienced Bad Saves. I don't know if it is because this is the first time I am playing the game on Windows 8.1 ( previously I was using XP and Vista with only 4 and 2 GB of Ram respectively )

I am now deleting the Options File each time it happens.

I only have 2 profiles ( one for story I am playing - the other with only 2 Saves )

In the story I am playing I usually wait till I have about 35 Saves then I transfer all but the last 5 to an archive folder and continue playing. The Bad Saves seem to occur even when there are very few Saves in the Save folder.

 

[Pieter Boelen]

In the story I am playing I usually wait till I have about 35 Saves then I transfer all but the last 5 to an archive folder and continue playing. The Bad Saves seem to occur even when there are very few Saves in the Save folder.

I have marked this as "Engine Limitation" because it IS a real issue, but I doubt any of the recent code changes has caused it. It is likely that this has plagued us since the stock game.
Even if it could be fixed, it would be a massively complicated undertaking to even try with no guarantee of success.

 

[salonikasurf]

I find that I get bad saves nearly every time I try to save when time is sped up x3 etc using the R key. if I then go back to normal game speed using the G key - Game Save often then works. Also often happens if you want to save soon after a location change. I think it is a recent thing as in only the year or two. I don't remember having any issues with this in Build 13 at all.

 

[Pieter Boelen]

@salonikasurf: What is your current modpack version?
Indeed @Levis' new Leveling System did have that as effect, but that was reported fixed again months back!

 

[salonikasurf]

Hi Pieter,
14 February B14 B4.

It may be a Windows 10 thing - although I did have issues on 8.1 as well. Sometimes it was good when an update came out and other times is was quite bad. So it is very hard to tell where the issues are.

 

[Ron foster]

I have been getting several "Bad Saves" with Beta 4.

When this happens I get this in the error.log

Happens in Win7 too... go figure.

Also, the save games does not warn when full and games "saved" after that point... are not

Over which only the coder gods have control... arrrggghhh matey

 

[ChezJfrey]

I have marked this as "Engine Limitation" because it IS a real issue, but I doubt any of the recent code changes has caused it. It is likely that this has plagued us since the stock game.
Even if it could be fixed, it would be a massively complicated undertaking to even try with no guarantee of success.

I would say your first two assertions are correct: It is an engine problem, and has plagued the game since the beginning. There is also a reason why this particular 'no memory' error at line 1048 appears in the log, along with the 'Bad Save', more often in standard save, vs. quick save; that NewInterfaceTexture is being called for each slot iteration to create a texture image for the save on the interface screen, calling MSG_SCRSHOT_READ message. This then requires the engine to allocate some memory to read and store that set of image bytes for display.

Each 32 bit process only gets a max of 2gb memory allocation. That memory is going to be consumed by the mod/program scripts, the models, textures, etc. As New Horizons has become more complex, offering more content, and more complicated, it undoubtedly initiates more memory transactions. Regardless of complexity, as one leaves locations, switches interfaces, etc., those memory allocations are deleted/restored accordingly -- they don't remain persistent forever. But here is where the 'engine limitation' comes in...the original executables are rife with memory leaks, missed/forgotten mem deletes, etc. When new memory is requested (e.g. the MSG_SCRSHOT_READ in this case), the amount of memory has to be a contiguous, unused block of the requires size, within the limitation of what Windows grants for a single 32 bit process. Memory leaks and Windows' own memory allocation algorithms, exacerbate memory fragmentation (not to be confused with disk fragmentation), which over time, severely curtails the size of contiguous, unused memory blocks and given that New Horizons now probably produces more allocate/deallocate transactions with more script activity, the memory is becoming quite fragmented.

What this means is that the game process might actually only be actively using 500mb, if the entire 2gb memory space it can theoretically use, is greatly fragmented, and it needs 100 mb allocation, but can't actually find a contiguous, unused block of that size within its given space, the memory allocation will fail. That is the scenario you are likely seeing here, because looking at the source code, that "no error" message is clearly in the old version of the GetSaveData function of the engine.

This leads to your 3rd statement: "Even if it could be fixed, it would be a massively complicated undertaking to even try with no guarantee of success."

I believe it has been fixed. The newest compilation employs a different GetSaveData, and also has corrected/removed hundreds of memory leaks, and also uses a different memory allocation algorithm, much improved over the Windows standard library; it is both quicker, more efficient, and greatly reduces memory fragmentation so this type of problem is much less likely to occur. New Horizons was also modified to work with it.

 

[ChezJfrey]

To clarify, when I state 'newest compilation' that is not 2.8 COAS, as that version still contained the many memory leaks, poorer memory allocation algorithm and other bugs; the version referred is my newest compilation. What is also interesting, that while there are a couple 'likes' for the above, to date, there is no interest in using my newest compilation and the reworked New Horizons enabling usage of this later and improved engine; so far only ERAS, a Russian mod, and Modder01 for CT have embraced it.

 

[LarryHookins]

Jeffrey, if I were still playing PotC I'd definitely be using your new engine. My latest work has been modding TEHO, and I haven't seen many engine problems there, and I haven't even played TEHO in weeks. I'm currently sailing the virtual Caribbean (in scale) in Vehicle Simulator.

Does your engine fix the need to run a frame rate limiter just to play PotC on a modern computer? That would be reason enough for anyone to give it a try.

Hook

 

[DeathDaisy]

To clarify, when I state 'newest compilation' that is not 2.8 COAS, as that version still contained the many memory leaks, poorer memory allocation algorithm and other bugs; the version referred is my newest compilation. What is also interesting, that while there are a couple 'likes' for the above, to date, there is no interest in using my newest compilation and the reworked New Horizons enabling usage of this later and improved engine; so far only ERAS, a Russian mod, and Modder01 for CT have embraced it.

It was very interesting to read about why the engine does what it does! I kinda get the theory, but I have no idea what to do or where to even start with applying the compilation Is it very complicated? Or is it just a switch of exes? Or somewhere in between? if it's not too far over my head I'd definitely be interested!

 

[pedrwyth]

I did download your reworking of New Horizons to use your newest compiled engine. It is naturally enough based on NH roughly as at Moddb release (ie latest formal public release March/April 2016) if I recall correctly. I would intend to try and add in the changes to bring the game up to WIP so it has the fixes/additions introduced since then and hopefully see more interest in using it emerge. I presumed I would have to winmerge compare your files with that (MODDB) to see which you had had to alter and then try and physically do an actual winmerge with the latest. However since you released it (July?) I have been busy with my summer activities and now travelling so won't be looking in earnest any further until the end of the year. If you have documented which files you had to rework it could help me a lot when I get back to it (unless someone else steps in in the meantime).

I also got somewhat sidetracked looking at your comments and workround for sky textures and then went looking for how it was done in later versions of storm games, one of the many fields I knew nothing about in the first place, and concluded it won't be me fixing that!

 

[ChezJfrey]

I did download your reworking of New Horizons to use your newest compiled engine. It is naturally enough based on NH roughly as at Moddb release (ie latest formal public release March/April 2016) if I recall correctly. I would intend to try and add in the changes to bring the game up to WIP so it has the fixes/additions introduced since then and hopefully see more interest in using it emerge. I presumed I would have to winmerge compare your files with that (MODDB) to see which you had had to alter and then try and physically do an actual winmerge with the latest. However since you released it (July?) I have been busy with my summer activities and now travelling so won't be looking in earnest any further until the end of the year. If you have documented which files you had to rework it could help me a lot when I get back to it (unless someone else steps in in the meantime).

I also got somewhat sidetracked looking at your comments and workround for sky textures and then went looking for how it was done in later versions of storm games, one of the many fields I knew nothing about in the first place, and concluded it won't be me fixing that!

Ha! So you gave it a look...that is encouraging. And yes, I picked up New Horizons awhile back and will not have the most current fixes/changes.

My intent was to preserve as much of the New Horizons experience as possible, while getting it functional in the new. While I have ascertained it at least launches a game and I can sail, fight, save/load, etc., given my experience with Modder01 and his efforts with CT, even in the much closer 2.5 engine, there are undoubtedly a few things that still need attention. Especially since I am unfamiliar with actual gameplay in POTC, NH and CT, so I have probably missed some minor details. My hope was to at get as close as possible, then someone like yourself and others much more familiar with New Horizons could pick up and attend to some of those remaining facets.

I will attempt to inform what I can remember.

First, script file changes. There is a handy piece of software that might prove helpful, at: FreeFileSync

It will help identify altered files, by selecting two directories, and changing the 'compare' filter to 'file content'. Then running the 'compare' feature, it will scan and identify changed files. The list of changed script files:

dialog.c
globals.c
InternalSettings.h
MAXIMUS_Functions.c
messages.h
NK.c
quicksave.c
sail_msg.h
seadogs.c
utils.c
BATTLE_INTERFACE\BattleInterface.c
BATTLE_INTERFACE\fast_reload_table.c
BATTLE_INTERFACE\Flags.c
BATTLE_INTERFACE\ispyglass.c
BATTLE_INTERFACE\LandInterface.c
BATTLE_INTERFACE\LogInterface.c
BATTLE_INTERFACE\msg_control.h
BATTLE_INTERFACE\reload_tables.c
BATTLE_INTERFACE\utils.c
CANNONS\Cannons_init.c
Characters\characters_init.c
Characters\CharacterUtilite.c
Characters\init\CommonQuest.c
Characters\init\FortCommandants.c
CONTROLS\controls.c
CONTROLS\init_pc.c
DIALOGS\ENGLISH\eng_officiant_dialog.h
INTERFACE\boal_map.c
INTERFACE\info_box.c
INTERFACE\interface.c
INTERFACE\itemsbox.c
INTERFACE\kam_articles.c
INTERFACE\kam_rations.c
INTERFACE\kam_selectmodel.c
INTERFACE\kam_selectsails.c
INTERFACE\kam_shipberthing.c
INTERFACE\kam_shiptransfer.c
INTERFACE\map.c
INTERFACE\messages.h
INTERFACE\NationRelation.c
INTERFACE\ransack_main.c
INTERFACE\save_load.c
INTERFACE\TradeBook.c
INTERFACE\transfer_crew.c
INTERFACE\transfer_main.c
INTERFACE\utilite.c
ISLANDS\islands.c
ISLANDS\Islands_init.c
Locations\locations_loader.c
Loc_ai\LAi_character.c
Loc_ai\LAi_defines.c
Loc_ai\LAi_events.c
Loc_ai\LAi_fightparams.c
Loc_ai\LAi_init.c
Loc_ai\LAi_login.c
Loc_ai\LAi_utils.c
SEA_AI\AIBalls.c
SEA_AI\AICameras.c
SEA_AI\AICannon.c
SEA_AI\AISea.c
SEA_AI\AIShip.c
SEA_AI\Script_Defines.h
SEA_AI\sea.c
Ships\Ships_init.c
Storyline\Bartolomeu\SL_utils.c
Storyline\GoldBug\SL_utils.c
Storyline\GoldBug\quests\quests_reaction.c
Storyline\standard\StartStoryline.c
Storyline\WoodesRogers\SL_utils.c
Storyline\WoodesRogers\quests\quests_reaction.c
Towns\Towntable.c
Weather\WhrSky.c
WorldMap\worldmap.c
WorldMap\worldmap_events.c
WorldMap\worldmap_globals.c
WorldMap\worldmap_init.c
WorldMap\worldmap_reload.c

Also, all the .ini files were slightly changed to disallow resizing of interface for wider resolution, most notable and significant changes were to rigging.ini for the way New Horizons handles flags. I think flags could be improved, because I have coded a feature to allow multi-row textures for flags, vs. the single row limitation, but I did not go so far as to making all the script file changes and combine the flag texture files to make use of that, and simply retained the current way they are handled in NH.

RESOURCE animation .ani files need additional definitions for more of the 'random' actions allowed. I simply copied an existing action multiple times (with the added 'number' used for random selection). Examples:

man_ab.ani
blaze.ani
Beatrice.ani
man.ani
beatrice_ab.ani

SKY changes: As you mention, the old version only has a _side for sky, but newer uses a right, left, back, forward texture. To accommodate POTC/NH, I added a feature in the engine that will look for a .useOld attribute and if noted, will presume _side is right, left, back, and forward, to create the cube texture for sky. I would think a better approach would be to create the additional sky textures and make use of the expanded feature of sky wrap-around.

Worldmap
rename islands.gm as mein.gm
add islands.gm locators for merchant, quest, island labels...this locator file probably needs some work, but I did the minimum to get it functional in the game. Will also need some script work to read/give the worldmap encounters the from/to locations the same way the current game works.

I also believe some texture and/or model files were moved to some different subdirs...can't remember them all, but the newer game does have some hardcoded paths that were changed from 2.0 to 2.8 version...I just can't remember them all and did not specifically document them.

 

[ChezJfrey]

Does your engine fix the need to run a frame rate limiter just to play PotC on a modern computer? That would be reason enough for anyone to give it a try.

Hook

I don't know the answer. I have never played POTC, so I do not have a decent basis for the problem in question. I also don't know if my particular PC is considered 'modern', since I run fairly archaic (relative to the bleeding edge types), LOL My most 'modern' PC is a laptop, Intel stock (no fancy video card), circa 2012

In any case, it appears to run OK in the preliminary tests I have run with no such limiter.

 

[Pieter Boelen]

I don't know the answer. I have never played POTC, so I do not have a decent basis for the problem in question. I also don't know if my particular PC is considered 'modern', since I run fairly archaic (relative to the bleeding edge types), LOL My most 'modern' PC is a laptop, Intel stock (no fancy video card), circa 2012

@LarryHookins is referring to this issue: "Solved" - Windows 10: Low Frame Rates Despite (BECAUSE OF!) Good Hardware
It is massively annoying and, as far as I'm aware, does not apply to CoAS.
So if indeed your engine fixes that for PotC, it is a massive win for you indeed!