Solved Improve Installer

[Pieter Boelen]

At the moment the manual work needed for that doesn't bother me so much; it is mainly the computer working on it and not me.
And all the checks have to be done anyway.

The main thing I would like to fix is to use 7za.exe to handle the file extraction process instead of what I use at the moment.
Haven't really found the time to do that, though. As it isn't technically broken, I've been distracted by other game issues.

 

[Pieter Boelen]

It took a bit of doing, but I'm officially back in the compiling business after my reinstall of Windows.
Here's to hoping it'll work as it should now!

 

[Levis]

Could I at least make one sugestion. Change the OUTPUT folder for the extracting to a folder inside the installation folder. I noticed I was running 2 instllations of the modpack at the same time and they we're overwriting each other in the OUTPUT folder cause they where both located in my download folder.

 

[Pieter Boelen]

That is indeed one of the stupid effects of the plugin I use.
Would be solved by using 7za.exe as then I could extract directly to the correct destination folder.
That'd be much more better.

 

[Pieter Boelen]

One thing that might be tricky with using 7za.exe in the installer is that I fear there would no longer be any progress updates during the extraction process.
How much of a problem do people think that would be? We may have to see if another solution exists anywhere.

 

[Grey Roger]

What does the installer do apart from extract the files and copy them to their relevant locations?

Obvious solution: if you want to run several installers at once, move each one to its own folder before running it.
Less obvious solution: 7-Zip is able to read the installer.exe and extract files itself. And you can tell it where to send the output.

 

[Pieter Boelen]

What does the installer do apart from extract the files and copy them to their relevant locations?

Writing from memory here, but the EXE does a whole lot of things:
- Check that you're installing to a folder that has the game already installed (classis mistake to not install the game at all or install to another folder )
- Checks that the main archive CRC checksum matches with the stored one to prevent installations from corrupted archives
- Unlocalize the game to ensure any different language game versions all work (allows for Russian, Spanish, German, French editions)
- Remove unused stock game files and clears out the entire PROGRAM and RESOURCE\INI folders to ensure a clean install of the mod files
- Copy a lot of stock game files to needed mod locations, most notably for the restructured deck model folders and the nation-specific location textures in RESOURCE\Textures\Locations
(this is to avoid having these files in the installer and reduces the overall size)
- Extract the main archive using the simple NSIS 7-Zip DLL, which unfortunately only extracts to the folder where the EXE is placed but does show progress during the installation
- Copy the modpack files from the installer directory to the game directory
- Do some more copying of modpack files so that those didn't have to be included several times in the archive file
- Ensure a SAVE folder is created and overwrite the options file with a clean one so that Config.exe works immediately

Then the process continues with installing the update files:
- Check that "Build Info.txt" is present, which should only be the case when the main archive was successfully installed first (classic mistake to install ONLY the update and skip on the majority of the game files)
- Install all update files directly to the main game folder
- Remove all sorts of recently depreciated files to ensure the game installation remains clear of useless files
- Peform some extra copying of files as required

If you look at the installer script, you'll see that the actions it performs are quite extensive....

Any sort of manual installion is just not a good idea under any circumstances whatsoever.
The installer must be allowed to do its job or the installation will be wrong. That's why we have it.

 

[Levis]

Have you tough about using the new nsis? Version 3 is out.

 

[Pieter Boelen]

Have you tough about using the new nsis? Version 3 is out.

I think I actually installed that before, but then I couldn't compile the installer with HM NIS Edit like I'm used to anymore, which was super annoying.
So I'd need an equivalent program to generate the installation scripts for me. And prior to that, I would need to know if it is worth the trouble.
The *only* thing I want to add to the installer is to have it extract a 7-Zip archive directly to the destination folder while showing progress.

 

[Pieter Boelen]

Yup, seems to be working. Will be included in the next EXE installer I make for testing purposes.

The above was my confirmation post that the CRC check in the installer works.
This was posted one week prior to the Beta 3.4 release, so indeed Beta 3.4 does also have the CRC check.

 

[Levis]

with the findings of @Kpax7 here http://www.piratesahoy.net/threads/runtime-error-engine-exe.23547/page-3#post-514145 wouldn't it be an wise idea to change the way the installer works?
I'd say let it extract everything to C:\POTC (or something like that) and then copy the files which are still needed from the install directory into this new folder and afterwards delete the install directory of the old potc install.
This way we know for sure it goes wel.
To be sure you could even say we do:
C:\POTC\Build_3_5
so you know for sure you wont overwrite something.

 

[Pieter Boelen]

I'd say let it extract everything to C:\POTC (or something like that)

How?

 

[Pieter Boelen]

I tried to use @Kpax7's suggestion for the installer EXE to make it extract directly to the installation folder and therefore skip the "extracting locally, then moving it" process.
And it seems that this here actually WORKS!

  ; Extract 7-Zip File
  SetOutPath "$INSTDIR"
  Nsis7z::ExtractWithDetails "$EXEDIR\b14_beta3_full.potc" "Extracting Main Modpack Files: %s..."

So this means we can:
- Skip a step in the process and hopefully speed it up
- Probably do away with the requirement to have the installer EXE outside the game folder

However, it also means I have to make a new version of the massive archive that everybody is going to have to re-download to make a clean install.
I'm afraid we're just going to have to deal with that, even if it is rather unfortunate.

On a happier note: ModDB supports files up to 4GB now (instead of 2GB),
which means we can include EVERYTHING in the archive and make the EXEs really small again for a while.

 

[Pieter Boelen]

Not quite having the success I hoped for yet. I successfully made a tiny 7-Zip archive extract to the correct folder.
But scaling it up to the full mod size, I'm getting just lots of 0 KB files instead of the actual content.
That probably sounds familiar to @Jason.

Perhaps the extracting DLL I'm using isn't compatible with the latest version of 7-Zip.
However, trying to zip all the required files with an older version gives me a "not enough RAM" error message or something and doesn't work either.

It seems that the old 7-Zip I need for the NSIS DLL cannot handle files much over 2GB or something.
My tests are continuing, but this is certainly annoying.

 

[Levis]

Else just make a part 1, 2 and 3.

 

[Pieter Boelen]

Else just make a part 1, 2 and 3.

I just tried re-zipping the Beta 3 archive files and that didn't work either. Same problem.
So I have yet to figure out how to make a 7-Zip archive that actually works again.

The problem clearly is NOT the size. Maybe the compression method.
Trying "LZMA" instead of "LZMA2" now, whatever the difference might be.

EDIT: Reading up on some stuff online, that might actually make a difference.

 

[Kpax7]

with the findings of @Kpax7 here http://www.piratesahoy.net/threads/runtime-error-engine-exe.23547/page-3#post-514145 wouldn't it be an wise idea to change the way the installer works?
I'd say let it extract everything to C:\POTC (or something like that) and then copy the files which are still needed from the install directory into this new folder and afterwards delete the install directory of the old potc install.
This way we know for sure it goes wel.
To be sure you could even say we do:
C:\POTC\Build_3_5
so you know for sure you wont overwrite something.

I don't think that's an option! Because we don't know if the User has enough space in that volume. We can't blindly write files to that folder. And If we wanted to do something like that we could use a better solution like the environmental variables like %temp% it's the best option. but once again we don't know if user has that capacity.

I tried to use @Kpax7's suggestion for the installer EXE to make it extract directly to the installation folder and therefore skip the "extracting locally, then moving it" process.
And it seems that this here actually WORKS!

  ; Extract 7-Zip File
  SetOutPath "$INSTDIR"
  Nsis7z::ExtractWithDetails "$EXEDIR\b14_beta3_full.potc" "Extracting Main Modpack Files: %s..."

So this means we can:
- Skip a step in the process and hopefully speed it up
- Probably do away with the requirement to have the installer EXE outside the game folder

Great! I'm happy that I could help solving the first puzzle

However, it also means I have to make a new version of the massive archive that everybody is going to have to re-download to make a clean install.
I'm afraid we're just going to have to deal with that, even if it is rather unfortunate.

On a happier note: ModDB supports files up to 4GB now (instead of 2GB),
which means we can include EVERYTHING in the archive and make the EXEs really small again for a while.

Nope! I think we could make the new patches "Backward compatible". You can add a small code in the NSI file to first look for the old Base archive and if it didn't exist look for the new one.
Something like this:

${If} ${FileExists} "$EXEDIR\b14_beta3_full.7z"
SetOutPath "$INSTDIR"
 Nsis7z::ExtractWithDetails "b14_beta3_full.7z" "Extracting Main Modpack Files: %s..."
   ${Else}
         ${If} ${FileExists} "$EXEDIR\b14_beta3_full_NEW .7z"  
             SetOutPath "$INSTDIR"
             Nsis7z::ExtractWithDetails "b14_beta3_full_NEW.7z" "Extracting Main Modpack Files: %s..."
         ${EndIf}
  ${Else}
  abort1:
  Banner::destroy
  Abort

Not quite having the success I hoped for yet. I successfully made a tiny 7-Zip archive extract to the correct folder.
But scaling it up to the full mod size, I'm getting just lots of 0 KB files instead of the actual content.
That probably sounds familiar to @Jason.

Perhaps the extracting DLL I'm using isn't compatible with the latest version of 7-Zip.
However, trying to zip all the required files with an older version gives me a "not enough RAM" error message or something and doesn't work either.

It seems that the old 7-Zip I need for the NSIS DLL cannot handle files much over 2GB or something.
My tests are continuing, but this is certainly annoying.

You mean you Can't make the 7zip or you Can't make the installer extract it correctly?
Have you tried to make the 7Z with other softwares out there?
It should be like this:

If you use the command line 7zip may I see the commands please?

Else just make a part 1, 2 and 3.

That's not option either! Ba go back to uploading and downloading multiple parts and the old headache

I just tried re-zipping the Beta 3 archive files and that didn't work either. Same problem.

Can you tell me "What didn't work?"
I mean the installer can't extract the 7zip or you can't 7zip them due to low RAM. Which part is the problem? The Creation of 7zip or EXTRACTING it?

 

[Pieter Boelen]

Nope! I think we could make the new patches "Backward compatible". You can add a small code in the NSI file to first look for the old Base archive and if it didn't exist look for the new one.
Something like this:

${If} ${FileExists} "$EXEDIR\b14_beta3_full.7z"
SetOutPath "$INSTDIR"
Nsis7z::ExtractWithDetails "b14_beta3_full.7z" "Extracting Main Modpack Files: %s..."
   ${Else}
         ${If} ${FileExists} "$EXEDIR\b14_beta3_full_NEW .7z" 
             SetOutPath "$INSTDIR"
             Nsis7z::ExtractWithDetails "b14_beta3_full_NEW.7z" "Extracting Main Modpack Files: %s..."
         ${EndIf}
  ${Else}
  abort1:
  Banner::destroy
  Abort

Could do.... If we keep the archives the same, but only zipped differently, the process can remain.
Will also require two independent CRC checks.

At the moment what I'm trying is to make a brand new archive that has EVERYTHING in it.
If I'm redoing it anyway, might as well do it right. Now the EXE I have is just 4 MB, so that is at least nice.
But that wouldn't work for backward compatibility at all.

Actually, I had a potentially good idea for the new method....
Have one Beta 4* unique file that is NOT in the archive itself, but is installed straight from the EXE.
That allows us to check if people tried to do a manual extract (even though I'm changing the file to "b14_beta4_full.potc") and give an error based on that.

* = If we're getting a new archive, I'm renaming the version!

Can you tell me "What didn't work?"
I mean the installer can't extract the 7zip or you can't 7zip them due to low RAM. Which part is the problem? The Creation of 7zip or EXTRACTING it?

I'm now testing with LZMA instead of LZMA2 compression; that seems to be working!
Still have some more waiting and fine-tuning to do, but we're getting there.

 

[Kpax7]

Could do.... If we keep the archives the same, but only zipped differently, the process can remain.
Will also require two independent CRC checks.

If the current work succeeds, you can release 2 versions. One that is compatible with old "b14_beta3_full.7z" and One with all files inside it. this way you keep all customers happy and doesn't take too much time...

Actually, I had a potentially good idea for the new method....
Have one Beta 4* unique file that is NOT in the archive itself, but is installed straight from the EXE.
That allows us to check if people tried to do a manual extract (even though I'm changing the file to "b14_beta4_full.potc") and give an error based on that.

That's a good idea. Like! Unless you figure out which unique file

I'm now testing with LZMA instead of LZMA2 compression; that seems to be working!
Still have some more waiting and fine-tuning to do, but we're getting there.

LOLz! You were supposed to use default LZMA, because all previous versions were based on it and were working
Anyway GREAT news! I love progressing in team work!

Always happy to help!

 

[Pieter Boelen]

Yep, it extracted properly this time! Tomorrow I'll continue so hopefully you'll see a first version based on the new system for testing later this week.
Thanks again.