It is possible to create a new nation but it is difficult! There is provision for two "Guest Nations". One of them is already used - Guest Nation 1 is America in "Revolutions" and "Napoleonic" periods, and may be Sweden in earlier periods if you have changed the setting in the "Options" menu to allow Sweden. Guest Nation 2 is currently unused and probably won't work if Guest Nation 1 is not active, which it won't be if you're playing one of the earlier periods and Sweden isn't enabled.
There is also a fake nation, PRIVATEER_NATION, which you could use. However, the "Smuggling for Thomas O'Reily" sidequest uses that for one of the smuggler ships.
The other problem is that no nation has single identifiable fleets. You can randomly encounter fleets belonging to any nation, both on worldmap and in direct sail, and the game will set up the fleet according to what sorts of ships are permitted to that nation in that period. So if you made ship types "CursedDutchman", "BlackPearl" and "QueenAnnesRevenge" available to your nation, you could encounter fleets randomly made up of one or more of each type of ship. You wouldn't encounter a fleet consisting of just the Flying Dutchman, you'd encounter a random fleet which could include Flying Dutchman and Queen Anne's Revenge, or two Black Pearls, for example. And PRIVATEER_NATION never spawns fleets on worldmap or during direct sail.
You might do better to set up a sidequest. Have a look at the code for the "Assassin" storyline ("PROGRAM\Storyline\Assassin\quests\quests_reaction.c"), specifically case "rocheship", in which Roche Brasiliano's ship is placed at a random island. You could probably copy and adapt that to put the Flying Dutchman at a random island. The main snag then is that either you're using character "Davy Jones" as the captain of the Flying Dutchman; or you're giving another Flying Dutchman to some other character. Either way is liable to lead to complications if you then play the "Elizabeth Shaw's Disappearance" sidequest as Davy Jones and the Flying Dutchman are used for the quest.