Brainstorming Enhanced Modding Support?

[Hammie]

I've been tinkering with improving modding support in the new engine: Modding Support · Discussion #96 · storm-devs/storm-engine

I don't know how Hook's Mod worked exactly but I'm interested to hear any ideas on how we can make the modding experience better.


EDIT: Split off from Notice - Hook's Mod for TEHO 1.7.0.ENG (09.04.20)

 

[Pieter Boelen]

I've been tinkering with improving modding support in the new engine: Modding Support · Discussion #96 · storm-devs/storm-engine

Had a chat with @kb31 on the Discord about that last night; but I'm sure you'll see that soon enough.

Honestly, I cannot quite oversee how all of that would end up functioning.
For example, where would something like "New Horizons" fit in?
It's basically a total conversion and changes virtually every file in a gazillion ways; many which are integrated with each other.
"Separate" mods tend to interfere with each other REAL quick; so dealing with them as individual files, with one taking precedence over another, could very easily go wrong.
Imagine two mods:
1. Replaces file A, B and C
2. Replaces A and B

End result: The game runs with files A and B from mod 2; plus file C from mod 1.
You now have a bugged game, because mod 1 doesn't work quite without it's own file A and B.


Alternative would be to not do it on file-basis; but on lines-in-a-file-basis.
This is what the very VERY old "ModHelper" (I think) for PotC used to do.
A "mod" was a single file, containing the names of the game code files to be changed, the unmodded section; and the modded section to replace it.
With one press of a button, you could swap the relevant sections of code between unmodded and modded version.
Of course you then still get the same problem I described above; but at least it'll be notably less likely.

But even "less likely" is still likely.
Doing exactly this is where my personal involvement with PiratesAhoy! started.
As soon as any mod reached even a relatively minor level of complexity, they quickly started modifying the same sections of code as other mods.
This immediately resulted in incompatibilities again; which I would then manually correct so the two mods would work together after all.
My first contribution on this forum was a ZIP file with the resulted merged files of doing this; saving other people the trouble of doing it themselves.


What to do also when different mods have pre-requisites? Or they start building on top of each other?
New Horizons is the cumulative result of exactly that happening for 18 years straight.
If we hadn't kept building on top of the previous mods, we could never have reached the level that we have now.


How to deal then with your example of a ship recolour?
Based on how PotC works, I'd say: just add a whole new ship with the new colour.
That's not so difficult; especially because the PotC Build ships collection is FAR easier to expand than in CoAS and TEHO.
(Reason: the PotC Build stopped using Ships.h #define numbers ages ago; just add a new ships_init.c entry and that's it.)

But for an even better solution (for adding ships), we should look at Virtual Sailor: Page Redirection
There EVERY file for one single ship is contained inside its own folder; so that includes models, textures and code.
Installing a new ship is as easy extracting one ZIP to your game folder and nothing else.
The game automatically loads every ship in the folder structure, so users don't need to modify anything at all.
Similarly, to remove a ship: just delete the folder.


When I compare the code of PotC Build with what I have seen in CoAS or TEHO, I see really MASSIVE differences.
And as far as I can see, a lot of the extremely valuable advances made in modding by the people here at PiratesAhoy! never made their way into the later games; or any other mods.
Which is a crying shame, because I honestly believe that in a great many ways, despite being for the oldest game, it also has the most advanced code.

So I would most strongly urge everyone wanting to make better Sea Dogs games in the future to have a very, very close look at everything that is already done in New Horizons.
I get the impression that most people who are familiar with PotC are unfamiliar with the later games; and those familiar with CoAS/TEHO are large unfamiliar with the PotC Build Mod.
But the way I see it, BY FAR the best possible results would be to take all the good things from all of those; and combine them together.
With a large focus being on the PotC Build code because, unless I'm VERY much mistaken,
that is where the most thorough reworkings of the PROGRAM structure can be found.

I imagine especially the rewritten Ships code system is something that would easily be desirable in any other Sea Dogs game.
The Console, Reinitialize and Items code are also very interesting; and that's just the tip of the iceberg.
As for features, since we now already have a Periods and Storylines mod and Translation support, I'd say it'd be madness to not make the best possible use of that.
Adding all of these to a new base game would immediately make modding support so much easier.


If I could describe my personal dream:
I would love a new Sea Dogs Storm Engine game that combined the best of all worlds.
It would have to be largely programmed from the "ground" up; probably starting from Stock PotC on the latest Storm Engine (which @Hammie is currently making a reality! ).

Directly copying all relevant code from the PotC Build is a bad idea, because while it might be complex and fancy; it is also quite messy and after 18 years it is very much due for a clean rewrite.
So we'd have to look across all the games in the series, find the most valuable features, and start re-adding them to our new base.

Then in my dream, ALL storylines and sidequests from New Horizons would eventually be included; in addition to the stories and quests from all other games.
Improvements to the base game and engine would therefore automatically affect every possible character.
And full storylines can easily be made modular; because effectively in PotC, they always are.

In the future then, you could have separate ship, item, location, character model and storyline downloads that you just drop on your base game.
"Mod packs" could then be a collection of those things that all fit together to tell, for example, a particular story.
Perhaps different time periods could be made modular that way too.

It starts to get especially interesting though when you get to modular game worlds.
Imagine having a base game, with regular Free Play in the Caribbean in the 17th century, but modular add-ons with:
- The Mediterranean Sea in Medieval times (including Worldmap, 3D Sailing Mode coastline, town models, character models, ship models, quests)
- The North American Coast in Civil War times (with plenty more engine-powered ships)
- Northern Europe, allowing us to FINALLY visit Amsterdam and play Hornblower-vs-Napoleon in the correct area
- Madagascar, Asia, etc.

My mind boggles when I think of all the possibilities.
But it all comes with a solid base.
Which we are very much in a position to create together!

 

[Hammie]

Thank you for the feedback and ideas!

For example, where would something like "New Horizons" fit in?

You would have at least two options:

• You move all modified files to a new "Mods/New Horizons" folder (but I think these are almost all files)
  
• Everything stays exactly the same

For the purpose of New Horizons, I think the main goal would be to allow the creation of simple 'sub-mods'. These (smaller) mods can also depend on other mods, so most mods will depend on New Horizons, and will be loaded after their dependencies.
This does not prevent conflicts from happening, as their is no magical solution to that, but it would for example allow easy removal of these mods, and - in most cases - will allow you to update New Horizons without merging file changes.

My goal with these changes is simply to make it easier for new modders to share their work. They can just upload the folder and anyone interested can try it out. I would also expect (and suggest) that we try to merge most of the good mods into New Horizons proper, but merging them after the mod has been "finished" might also make New Horizons more stable.

But, just to be clear, I just want to provide more options, not force you down a particular path.

@kb31's idea for using event handlers and hooks would also allow mods to make changes without actually modifying the code. If I understand correctly it should allow you to create a config file that tells to game to execute specific scripts at the start or end of existing functions. Something like: "Execute 'MyFunction()' at the beginning of 'Main()'", which would then insert a call to 'MyFunction()' at the very start of the 'Main()' function, thus executing it as one of the very first things as the game loads.

As for adding ships, items and other asset-like content, I think we all agree that drop-in support would be ideal. Not sure how to achieve this yet though. Should we move this kind of feature to the engine, or give scripts the ability to inspect mods and configs instead?

So in summary, still lots to think about and figure out.

My mind boggles when I think of all the possibilities.
But it all comes with a solid base.

I think I share most of those dreams, but I'll stick to getting New Horizons to work and fixing bugs for now

 

[Pieter Boelen]

But, just to be clear, I just want to provide more options, not force you down a particular path.

I understand.
I also just posted what I did for consideration.
In the end, we're going to have to do things as they come; and adapt where relevant.

@kb31's idea for using event handlers and hooks would also allow mods to make changes without actually modifying the code.

I've heard of constructs like that.
But for now, I can't even remotely wrap my head around them.
Too fancy for my simple brain...

I think I share most of those dreams, but I'll stick to getting New Horizons to work and fixing bugs for now

Wise.

We can dream.
Maybe plan and prepare a bit here and there.
But the only way of getting anywhere is by doing the basics first.

In other words: you've got your priorities exactly right.

 

[Pieter Boelen]

As for adding ships, items and other asset-like content, I think we all agree that drop-in support would be ideal. Not sure how to achieve this yet though. Should we move this kind of feature to the engine, or give scripts the ability to inspect mods and configs instead?

It's actually already possible. @Inez Dias made a proof of concept of that for location init entries:

// Inez Dias -->
extern int InitModLocations();
object LocDirectory;

int LocationInitModLocations(int n)
{
   DeleteAttribute(&LocDirectory, "");

   ref pc = GetMainCharacter(); // KK
   LocDirectory.dir = "Program\\Locations\\init\\mods";
   LocDirectory.mask = "*.c";
   CreateEntity(&LocDirectory,"FINDFILESINTODIRECTORY");
   DeleteClass(&LocDirectory);
   aref arList;
   makearef(arList, LocDirectory.filelist);
   for(int i = 0; i < GetAttributesNum(arList); i++)
   {
       aref arCur = GetAttributeN(arList, i);
       string fname = GetAttributeValue(arCur);
       string fullname = "locations\\init\\mods\\"+fname; 
       if(LoadSegment(fullname))
       {
           pc.numlocations = n; //Hack because we can't pass parameters to extern functions
           n = InitModLocations();
           DeleteAttribute(pc, "numlocations");
           UnloadSegment(fullname);
       }
       else
       {
           Log_SetStringToLog("There's an error in the location init code in" + " " + fname);
       }
   }
   return n; // KK
}
// Inez Dias <--

And then that mentioned "mods" subfolder can contain files like these:

/*
Inez Dias -->
New locations can easily be added by just dropping a file into this directory, without any further editing of locations_init.c. Make sure the below code is in the file.
*/

int InitModLocations()
{
   int n = GetMainCharacter().numlocations;

   // ...
   // increase n by 1 for each new location.

   return n;
}

Adapting that to ships_init.c entries should be fairly straightforward, I expect.

No changes to the game engine code needed at all.
Stock PotC Storm 2.0 could do it.

I only just remembered about this; and it's yet further proof for my earlier point:
The PotC Build Mod code is really already full of hidden pirate gold nuggets with solutions to all sorts of stuff.