The difference between Dolphin's game IDs and GameTDB's game IDs
is that GameTDB uses four characters for non-disc titles, whereas
Dolphin uses six characters for all titles.
This fixes:
- TitleDatabase considering Datel discs to be NHL Hitz 2002
- Gecko code downloading not working for discs with IDs starting with P
- Cover downloading mixing up discs with channels (e.g. Mario Kart Wii
and Mario Kart Channel) and making extra HTTP requests. (Android was
actually doing a better job at this than DolphinQt!)
These country codes have the unfortunate property that they are used
by Wii disc games in two different regions. We already correct for this
in VolumeGC::GetCountry and VolumeWii::GetCountry, so this commit
shouldn't really have any effect on how the game list behaves,
but it will be useful if we in the future would want to call
CountrySwitch directly without having extra code in the caller for
handling region weirdness.
The header of a Wii disc can be read from two places: The
unencrypted area at the beginning of the disc, or the beginning of
the game partition. The two copies are usually identical (except
for 0x60 and 0x61), but there are exceptions. For most of Dolphin's
history, we have been reading from the header inside the game
partition when getting metadata. This was however not the case
starting with 4.0-4901 and ending with 5.0-3762. This commit once
again makes Dolphin read metadata from the unencrypted header,
because of the following reasons that I recently was informed about:
- The "pink fish" disc has the game ID 410E01 in the unencrypted
header but the placeholder game ID RELSAB in the partition header.
- The revisions of some games differ between the two headers,
with the unencrypted one making more sense.
(See https://bugs.dolphin-emu.org/issues/11387)
For better or worse, this also means that sloppily hacked games where
only the game ID in the unencrypted header has been changed now will
use that modified game ID. And unlike with the partition header,
there is no signing or hashing that can tell us whether the
unencrypted header has been modified by someone other than Nintendo.
ChunkFile doesn't use any of the file utilities, so we can drop these
headers to avoid pulling in unnecessary dependencies. This also
uncovered a few indirect inclusions.
We can just use std::any_of here to collapse the checking code down to a
single assignment as opposed to a loop. This also slightly improves on
the existing code, as this won't continue to iterate through the cluster
metadata if an entry that's non-zero is encountered.
These disc images are only used on dev units and not retail units.
There are two important differences compared to normal Wii disc images:
- The data starts 0x8000 bytes into each partition instead of 0x20000
- The data of a partition is stored unencrypted and contains no hashes
Our old implementation was just guesswork and doesn't work at all.
According to testing by GerbilSoft, this commit's implementation
is able to read and extract files in the filesystem correctly,
but the tested game still isn't able to boot. (It's thanks to their
info about unencrypted disc images that I was able to make this commit.)
Since all FS access will go through the new FS interface (PR #6421)
in order to keep track of metadata properly, there is no need to return
absolute paths anymore.
In fact, returning host paths is a roadblock to using the FS interface.
This starts the migration work by adding a way to get paths that are
relative to the Wii NAND instead of always getting absolute paths
on the host FS.
To prepare for future changes, this commit also makes returned paths
canonical by removing the trailing slash when it's unneeded.
Eventually, once everything has been migrated to the new interface,
we can remove the "from" parameter.
The original reason I wanted to do this was so that we can replace
the Android-specific code with this in the future, but of course,
just deduplicating between DolphinWX and DolphinQt2 is nice too.
Fixes:
- DolphinQt2 showing the wrong size for split WBFS disc images.
- DolphinQt2 being case sensitive when checking if a file is a DOL/ELF.
- DolphinQt2 not detecting when a Wii banner has become available
after the game list cache was created.
Removes:
- DolphinWX's ability to load PNGs as custom banners. But it was
already rather broken (see https://bugs.dolphin-emu.org/issues/10365
and https://bugs.dolphin-emu.org/issues/10366). The reason I removed
this was because PNG decoding relied on wx code and we don't have any
good non-wx/Qt code for loading PNG files right now (let's not use
SOIL), but we should be able to use libpng directly to implement PNG
loading in the future.
- DolphinQt2's ability to ignore a cached game if the last modified
time differs. We currently don't have a non-wx/Qt way to get the time.
We shouldn't try to create folder names that contain characters
such as : or / since they are forbidden or have special meanings.
(No officially released disc uses such characters, though.)
Use std::string(cstring, strnlen(cstring, max_length)) instead of
trying to remove extra null characters manually, which is a bit
ugly and error prone.
And indeed, the original code contained a bug which would cause
extra NULLs to not be removed at all if the string did not
end with a NULL -- causing issues down the road when constructing
paths for sub-entries.
This removes the hack that enables directly booting from WADs
without installing them first for the following reasons:
1. It makes the NAND content handling much more complicated than what
it should be and makes future changes like permissions or booting
NAND titles without a WAD more annoying to implement.
Because of this hack, we needed an extra level of abstraction
(NANDContent*) which has to read tons of things from the NAND, even
most of the time it's useless. This in turn forces us to have
caching, which is known to break titles and requires manual cache
invalidations. Annoying and error prone.
2. It prevents the WAD boot code from being easily accurate. With this
change, we can simply reuse the existing launch code, and ask IOS
to launch the title from the NAND.
3. The hack did not work that well since it did not cover a lot of ES
commands. And it works even less since the ES accuracy fixes.
This results in Dolphin returning inconsistent results: a
lot of the ES "DI" commands will just fail because the active title
is not installed on the NAND. uid.sys is not changed, etc.
And I'm not even talking about FS stuff -- where this would still
totally fail, unless we add even more unnecessary hacks.
This is not just theoretical -- the system menu and the Wii Shop are
known to behave strangely because the hack damages the NAND
structure, and we've already had several users report issues.
This commit makes it so WADs are always installed prior to launching.
A future commit will remove any code that was there only for the hack.
This updates the maker data to (mostly) mirror that of the Wiki:
https://wiki.dolphin-emu.org/index.php?title=GameIDs
Only maker ids from that page are now included in Dolphin. This
means no homebrew/unofficial makers.
Also, separate multiple maker names with a slash
Some of my WiiWare games does not have a maker :
- Blue's Journey : EAFPJ8
- Magician Lord : EACPJ8
- The King of Fighters '94 : EAGPJ8
- The Last Ninja : C9XPGX
- World Games : C9ZPGX
I noticed the Strong Bad games, FAST - Racing League, and Tetris Party
were lacking info in the game lists' maker column.
This adds the information based on the games' MakerID.
This adds a WiiWad constructor that takes a BlobReader, so that the
class can be used with more than just files from the host filesystem.
Required for using WiiWad with WADs from update partitions.
There were two problems with this:
1. If the starting offset was beyond the end of the disc,
we would dereference an invalid iterator.
2. The data beyond the end of the disc was non-deterministic.
Having DiscContents with size 0 would mean that some DiscContents
might not get added to the std::set because of them comparing
identically to another DiscContent.
This replaces an older piece of code in WriteDirectory that ensures
that no two files have the same starting offset. (We now care about
the ending offset, not the starting offset. The new solution both
ensures that no two files have the same ending offset and that no
two files have the same starting offset.)
For instance, we don't want to show TGC files that might be
inside the /files/ directory of a GameCube DirectoryBlob,
and we don't want to show the /sys/main.dol files for extra
partitions of Wii DirectoryBlobs.
Now it's clearer that SetDOL depends on SetApploader
and BuildFST depends on SetDOL.
As a side note, we now load the DOL even if there's
no apploader. (I don't think it matters whether we
do it, but it was easier to implement this way.)
This lets VolumeDirectory/DirectoryBlob skip implementing
various volume functions like GetGameID, GetBanner, etc.
It also lets us view extracted discs in the game list.
This ends up breaking the boot process for Wii
DirectoryBlobs due to workarounds being removed from the
boot process, but that will be fixed later by adding
proper DirectoryBlob support for things like TMDs.
We now expect the directories to be laid out in a certain
format (based on the format that WIT uses) instead of requiring
the user to set the DVD root and apploader path settings.
This is useful for blob types that store Wii data unencrypted
(such as WIA and discs extracted to directories) so that
we don't have to waste CPU time encrypting in the blob code
just to decrypt right afterwards in the volume code.
According to http://scanlines16.com/en/blog-3/retro-gaming/game-cube/gamecube-korean-master-list/,
Korean GC releases use the following country codes:
- E or W for games in English
- K for games in Korean
- Unknown value for games in Japanese (my guess is that they might
have made the discs bit-for-bit identical to Japanese releases
because the regions of these games are already set to NTSC-J)
As far as I know, the GC has no Taiwanese releases, which is what
the W country code is used for on the Wii. But I could be wrong.
A small note: The country_byte == 'K' check in the code isn't
actually necessary as long as RegionSwitchGC returns NTSC_J
for 'K', but I thought it would be better to not rely on that.
The county code isn't 100% reliable for detecting the region.
For instance, some games released in Korea have the country
code E even though they're region-locked to NTSC-J consoles.
This commit makes the GC disc region detection match the Wii
disc region detection (apart from the region value being in
a different place on the disc).
Some code was calling more than one of these functions in a row
(in particular, FileUtil.cpp itself did it a lot...), which is
a waste since it's possible to call stat a single time and then
read all three values from the stat struct. This commit adds a
File::FileInfo class that calls stat once on construction and
then lets Exists/IsDirectory/GetSize be executed very quickly.
The performance improvement mostly matters for functions that
can be handling a lot of files, such as File::ScanDirectoryTree.
I've also done some cleanup in code that uses these functions.
For instance, some code had checks like !Exists() || !IsDirectory(),
which is functionally equivalent to !IsDirectory(), and some
code was using File::GetSize even though there was an IOFile
object that the code could call GetSize on.
Before, if you extracted a directory like /map/Final/Release/,
Dolphin would create the nested folders map, Final and Release
in the output directory and put the files in Release instead of
just putting the files directly in the output directory.
This makes it possible to catch errors earlier so that file systems
simply fail to load instead of behaving oddly. It also makes it possible
to check for errors that weren't checkable before, like the end of a
directory being after the end of the parent directory.
Instead of expecting callers to know how the size of directory file infos
relates to which files are in which directories, filesystems now offer a
GetRoot() method, and file infos offer a way to get their children. As
a bonus, m_FileInfoVector no longer has to be created and kept around
in RAM. Only the file info objects that actually are used are created.
Not initializing until the filesystem is used is good when
a filesystem is constructed and then never used, but nobody does that.
This simplifies the code a little and lets all methods be const.
Instead of using lots of small scattered reads to read the FST,
only one big read is used, which is more efficient.
This also means that the FST only allocates memory once and stores all
strings close to each other - good for the CPU cache. The file info
objects use pointers to this FST memory of containing data themselves.
Keeping around the big m_FileInfoVector containing objects with only
pointers is a bit unnecessary, but that will be fixed soon.
Instead of calling GetPathFromFSTOffset for every file info, FindFileInfo
now only looks at names in directories that are included in the path.
For the common case of searching for "opening.bnr", this means that
only root-level files and directories have to be searched through.
Some callers already have the file info, making the relatively slow
FindFileInfo calls unnecessary. Callers that didn't have the file info
will now need to call FindFileInfo on their own.
Some callers (i.e. ISOProperties) don't want the full path, so giving them
it is unnecessary. Those that do want it can use GetPathFromFSTOffset.
Not storing full paths everywhere also saves a small bit of RAM and is
necessary for a later commit. The code isn't especially pretty right now
(callers need to use FST offsets...) but it'll become better later.
GC/Wii filesystem internals shouldn't be exposed to other classes.
This change isn't especially useful by itself, but it opens up the
way for some neat stuff in the following commits.
You may want to read the PR #2047 comments before reading this.
Dolphin attempts to support an unencrypted type of Wii discs
that apparently is identified by a 4-byte integer at 0x60
being non-zero. I don't know what discs (if any) would be
using that format, so I haven't been able to test Dolphin's
support for it, but it has probably been broken for a while.
The old implementation is very short but also strange.
In CreateVolumeFromFilename, we read a 4-byte integer from
0x60, and if it's non-zero, we create a CVolumeGC object
instead of a CVolumeWiiCrypted object. This might seem like
it makes no sense, but it presumably worked in the past
because IsVolumeWiiDisc used to check the volume type by
reading the magic word for Wii straight from the disc,
meaning that CVolumeGC objects representing unencrypted Wii
discs would be treated as Wii discs by pretty much all of
Dolphin's code except for the volume implementation code.
(It wasn't possible to simply use CVolumeWiiCrypted, because
that class only handled encrypted discs, like the name says.)
However, that stopped working as intended because of ace0607.
And furthermore, bb93336 made it even more broken by making
parts of Dolphin expect that data read from Wii discs needed
to be decrypted (rather than the volume implementation
implicitly deciding whether to decrypt when Read was called).
Disclaimer: Like I said before, I haven't been able to test
any of this because I don't have any discs that use this
unencrypted Wii disc format, so this is all theoretical.
Later, PR #2047 tried to remove Dolphin's support for
the unencrypted Wii disc format because seemingly no
discs used it, but the PR got closed without being merged.
At the end of that PR, I said that I would make a new PR
with a better implementation for the format after PR #2353
was merged. Now that PR #2353 is merged (two years later...)
and PR #5521 is merged, the new implementation was easy to
make, and here it is!
Untested.
Normal users don't care about it. In fact, people care so
little about it that the Wii implementation of it was broken
starting from when it was implemented (eb65601) to 7 years
later (e0a47c1), apparently without anyone reporting it.
This makes the interface slightly cleaner and a bit more consistent
with the other getters. Still not fully the same, since the others
don't really handle failures with std::optional; but at least the
value is returned by value now, as opposed to having the function
take a pointer to a u64.
This commit moves the write function to where it should be (IOS),
especially when ES::ImportTicket() is the only place to use it.
Prevents misusing the ticket import function, and removes one unsafe
direct write to the NAND that does not go through IOS.
This also fixes the destination path: the session root is the one which
should be used for determining the ticket path, not the configured one.
The whole NANDContentLoader stuff is truly awful and will be removed
as soon as possible.
For now, this fixes a bug that was exposed by std::optional::operator*.
gcc complains that the printf %x formatting instruction expects an
'unsigned int' but we pass a 'size_t'. We add the 'z' length formatting
specifier used for 'size_t'
This file is pretty small now that it doesn't handle Wii
partitions anymore, so let's move its contents to Volume.cpp.
This is also more consistent with how blob creation works.
This happened to work without any problems because the only way for a
file system to be invalid was to not have the right GC/Wii magic word
in the unencrypted area, and a volume could not be created without
having the right GC/Wii magic word there. Now that file systems read
the magic word from a partition instead, a fix is needed.
I replaced m_OffsetShift with m_Wii in bb93336 to support
the decrypt parameter for read functions. Doing that is no
longer necessary, so m_offset_shift is now used like before.
By removing mutable state in VolumeWiiCrypted, this change makes
partition-related code simpler. It also gets rid of other ugly things,
like ISOProperties's "over 9000" loop that creates a list of
partitions by trying possible combinations, and DiscScrubber's
volume swapping that recreates the entire volume when it needs to
change partition.
Direct access to the WAD bytes is required to read contents with proper
padding data (since they can sometimes end up being outside of the
data app section). Allowing the whole buffer to be accessed directly
would be error prone, so this commit adds GetContent() to WiiWAD
for getting raw content data by index.
Just like DeleteTitle, Using CNANDContentManager is overkill,
inefficient and useless. And it results in a few failures in
situations where a delete should just always work.
But here it gets bonus points, because it manages to actually use
the TMD for deleting contents, when IOS does none of that and just
deletes files ending with .app in the title content directory. :)
Fixes warning:
```
Source/Core/DiscIO/NANDImporter.cpp:55:17: warning: format specifies type 'unsigned long' but the argument has type 'u64' (aka 'unsigned long long') [-Wformat]
file.GetSize(), NAND_BIN_SIZE);
^~~~~~~~~~~~~~
1 warning generated.
```
This will be required for permission checks in the future.
Note that this is only for the PPC as we do not have actual processes.
Keeping track of other modules' UIDs/GIDs is virtually useless anyway.
UID/GID changes are implemented in the following functions:
* ES_Launch
* ES_DIVerify
ES_SetUid is not implemented yet because it'd need further changes.
Fixes file ordering in games that use ASCII characters between lowercase
'z' and uppercase 'A' (underscores).
MySims Kingdom has the files "terrainLightMapTinted.shader",
"terrainLightMapTintedGrid.shader", and
"terrainLightMapTinted_no_shadow.shader". In lowercase,
"terrainLightMapTinted_no_shadow.shader" comes before
"terrainLightMapTinted.shader" and "terrainLightMapTintedGrid.shader",
which is invalid.
The size field in FSTEntry contains the total amount of children, not
including the parent, but the parent needs to be included.
VolumeDirectory: Fix off-by-one in entry count calculation
The size field in FSTEntry contains the total amount of children, not
including the parent, but the parent needs to be included.
Fixes warnings:
```
../Source/Core/DiscIO/VolumeGC.cpp: In member function 'virtual u8 DiscIO::CVolumeGC::GetDiscNumber() const':
../Source/Core/DiscIO/VolumeGC.cpp:178:10: error: 'disc_number' may be used uninitialized in this function [-Werror=maybe-uninitialized]
return disc_number;
^
../Source/Core/DiscIO/VolumeWiiCrypted.cpp: In member function 'virtual u8 DiscIO::CVolumeWiiCrypted::GetDiscNumber() const':
../Source/Core/DiscIO/VolumeWiiCrypted.cpp:258:10: error: 'disc_number' may be used uninitialized in this function [-Werror=maybe-uninitialized]
return disc_number;
^
../Source/Core/DiscIO/VolumeWiiCrypted.cpp: In member function 'virtual IOS::ES::TMDReader DiscIO::CVolumeWiiCrypted::GetTMD() const':
../Source/Core/DiscIO/VolumeWiiCrypted.cpp:123:20: error: 'tmd_address' may be used uninitialized in this function [-Werror=maybe-uninitialized]
tmd_address <<= 2;
^
```
Dolphin assumes that content 0 is opening.bnr, without checking
whether content 0 exists or if it is even supposed to be there (it's
only there for channels). This results in sometimes reading garbage.
This adds a check to only try to read names from content 0's header
if the title is a channel (channel, system channel or game channel).
The Tools > Load System Menu option displays the version of the
installed Wii Menu. This commit changes the way we display that
version, like so: "Load System Menu 514P" -> "Load System Menu 4.3E"
The numbers are from http://wiibrew.org/wiki/System_Menu
5.0-2712 made ES's code for setting the game ID use the
title ID converted to hex (except for disc titles) instead
of using a 6-char game ID like before. Then, 5.0-2830 made
us use that code even when loading game INIs. This breaks the
expectations of both users and the game INIs we ship with.
This commit makes Dolphin use 6-char game IDs for all
titles (unless the 6-char ID would contain unprintable
characters, which is the case with e.g. the Wii Menu).
I'm also putting unprintability checks in VolumeWad
for consistency.
Advantages:
* Simpler code in general
* No extra volume objects created
* Now actually notices if the disc or partition gets
changed while the core is running
* No longer picks up on disc access done by the GUI
(it used to do so as long as the core was running)
* Gets rid of a Core dependency in DiscIO
There are two performance disadvantages:
* FileMonitor is now a bit slower when used with VolumeDirectory
because FileMonitor now always uses the FileSystemGCWii code
for finding filenames instead of VolumeDirectory finding the
filename on its own and directly hooking into FileMonitor.
But this isn't such a big deal, because it's happening on the
DVD thread, and my currently unmerged file system PR will make
FileSystemGCWii's file finding code about as fast as
VolumeDirectory's.
* FileMonitor's creation of the file system object is now
done on the CPU thread instead of the DVD thread, and
it will be done even if FileMonitor logging is disabled.
This will be fixed in the next commit.
There's no point in creating a volume without a blob,
since essentially all the functionality of a volume
requires a blob to be used.
Also, VolumeCreator doesn't support creating volumes
without blobs (it can't even figure out the volume type
unless it gets a blob), so it's currently impossible
for a volume to be created without a blob.
Most of the Volume code was written before this
convenience function was added. Let's use it more.
Also deleting m_pReader nullptr checks that are
unnecessary because of Read (which ReadSwapped calls)
already having a nullptr check.
This moves all the byte swapping utilities into a header named Swap.h.
A dedicated header is much more preferable here due to the size of the
code itself. In general usage throughout the codebase, CommonFuncs.h was
generally only included for these functions anyway. These being in their
own header avoids dumping the lesser used utilities into scope. As well
as providing a localized area for more utilities related to byte
swapping in the future (should they be needed). This also makes it nicer
to identify which files depend on the byte swapping utilities in
particular.
Since this is a completely new header, moving the code uncovered a few
indirect includes, as well as making some other inclusions unnecessary.
This changes ES to keep track of the active title properly,
just like IOS:
* It is NOT changed on resource manager open/close.
* It is reset on IOS reload.
* It is changed by ES_DIVerify and ES_Launch.
IOS stores the active title in a structure like this:
struct ESTitleContext
{
Ticket* ticket;
TMD* tmd;
u32 active;
};
With this commit, we also do keep the Ticket and TMD around. This
makes some of the DI ioctlvs (which return data about the current
active title) trivial to implement in the future.
This fixes the System Menu not being able to see update partitions
and also allows us to change Dolphin's active game info in the future.
We already have a TMDReader, so let's actually use it.
And move ESFormats to IOS::ES, since it's definitely part of IOS.
This adds a DiscIO dependency on Core which will be fixed in a
follow-up PR.
This prevents Dolphin from writing to /sys/uid.sys (on the host; root
partition) when installing a WAD before starting emulation, because
the session root is not initialized at that moment.
Incidentally, this also gets rid of a singleton.
This fixes a regression from 5.0-1556, but I don't know why
the regression occurred or why this fixes it. (Many games
failed to fully boot - I tried Metroid Prime and Twilight
Princess (both GC), and they never got to the title screen.)
These two functions load either a signed ticket or a raw ticket from the
emulated NAND.
The ticket signature skip is refactored out of the ticket writing in
order to be usable by the raw ticket reading function.
Refactor the existing DiscIO::AddTicket to not require the caller to
pass the requested title ID. We do not have the title ID in the ES case,
and it needs to be extracted from the ticket. Since this is always a
safe operation to do (title ID is always in the ticket), the
implementation is made default.
Instead of needing different switch cases for
converting countries to regions in multiple places,
we now only need a single country-to-region switch case
(in DiscIO/Enums.cpp), and we get a nice Region type.
We can't rely on the OS returning files and directories
in a deterministic order, so we should sort them on our own
if we want VolumeDirectory to work for movies and netplay.
These disc images are encrypted and signed using a different set of keys.
We only care about the master key, so we check the signature issuer. If
it matches the debug issuer, then we'll use the RVT-R key. Otherwise,
the previous set of common and Korean keys are used.
This would previously fail to compile when included in files that do not
include FileUtil.h due to lack of a type declaration.
This moves the constructor and destructor into the cpp file in order to
satisfy the requirements of unique_ptr construction and deletion. That is,
unique_ptr requires a concrete type at the point of construction and
destruction. If the constructor or destructor is left in the header, then
at the point of construction or destruction, IOFile will still be
considered an incomplete type, as unique_ptr's deleter will still only be
able to see the forward declaration, which it can't use.
These functions don't actually depend on any state from the class
instance, so they don't really belong in the header, and are just
an implementation detail.
At first there weren't many enums in Volume.h, but the number has been
growing, and I'm planning to add one more for regions. To not make
Volume.h too large, and to avoid needing to include Volume.h in code
that doesn't use volume objects, I'm moving the enums to a new file.
I'm also turning them into enum classes while I'm at it.
If banner loading fails once, it will very likely fail again.
Setting m_banner_loaded to true when banner loading fails
prevents LoadBannerFile from wasting time if it's called again.
Banner loading requires loading the file system, which takes a
noticeable amount of time, so this matters.
DriveReader::m_size was never initialized which was indirectly
causing CGameListCtrl to crash Dolphin when it tried to insert a
character at a negative index in a string.
Reading one sector at a time is very inefficient and appears to
be causing timing issues during boot so SectorReader has been
enhanced to support batching.
SectorReader has been given a working cache system.
instead, leave all the management with the NANDContentLoader.
for file data (directly on the NAND), this opens the file on-demand and
returns the requested chunk when asked for it.
for on-the-fly decrypted WAD data, we just keep the decoded buffer in
memory, like we've done before - except that we don't give away any objects
we don't want to.
JosJuice