Now it's clearer that SetDOL depends on SetApploader
and BuildFST depends on SetDOL.
As a side note, we now load the DOL even if there's
no apploader. (I don't think it matters whether we
do it, but it was easier to implement this way.)
This lets VolumeDirectory/DirectoryBlob skip implementing
various volume functions like GetGameID, GetBanner, etc.
It also lets us view extracted discs in the game list.
This ends up breaking the boot process for Wii
DirectoryBlobs due to workarounds being removed from the
boot process, but that will be fixed later by adding
proper DirectoryBlob support for things like TMDs.
We now expect the directories to be laid out in a certain
format (based on the format that WIT uses) instead of requiring
the user to set the DVD root and apploader path settings.
This is useful for blob types that store Wii data unencrypted
(such as WIA and discs extracted to directories) so that
we don't have to waste CPU time encrypting in the blob code
just to decrypt right afterwards in the volume code.
According to http://scanlines16.com/en/blog-3/retro-gaming/game-cube/gamecube-korean-master-list/,
Korean GC releases use the following country codes:
- E or W for games in English
- K for games in Korean
- Unknown value for games in Japanese (my guess is that they might
have made the discs bit-for-bit identical to Japanese releases
because the regions of these games are already set to NTSC-J)
As far as I know, the GC has no Taiwanese releases, which is what
the W country code is used for on the Wii. But I could be wrong.
A small note: The country_byte == 'K' check in the code isn't
actually necessary as long as RegionSwitchGC returns NTSC_J
for 'K', but I thought it would be better to not rely on that.
The county code isn't 100% reliable for detecting the region.
For instance, some games released in Korea have the country
code E even though they're region-locked to NTSC-J consoles.
This commit makes the GC disc region detection match the Wii
disc region detection (apart from the region value being in
a different place on the disc).
Some code was calling more than one of these functions in a row
(in particular, FileUtil.cpp itself did it a lot...), which is
a waste since it's possible to call stat a single time and then
read all three values from the stat struct. This commit adds a
File::FileInfo class that calls stat once on construction and
then lets Exists/IsDirectory/GetSize be executed very quickly.
The performance improvement mostly matters for functions that
can be handling a lot of files, such as File::ScanDirectoryTree.
I've also done some cleanup in code that uses these functions.
For instance, some code had checks like !Exists() || !IsDirectory(),
which is functionally equivalent to !IsDirectory(), and some
code was using File::GetSize even though there was an IOFile
object that the code could call GetSize on.
Before, if you extracted a directory like /map/Final/Release/,
Dolphin would create the nested folders map, Final and Release
in the output directory and put the files in Release instead of
just putting the files directly in the output directory.
This makes it possible to catch errors earlier so that file systems
simply fail to load instead of behaving oddly. It also makes it possible
to check for errors that weren't checkable before, like the end of a
directory being after the end of the parent directory.
Instead of expecting callers to know how the size of directory file infos
relates to which files are in which directories, filesystems now offer a
GetRoot() method, and file infos offer a way to get their children. As
a bonus, m_FileInfoVector no longer has to be created and kept around
in RAM. Only the file info objects that actually are used are created.
Not initializing until the filesystem is used is good when
a filesystem is constructed and then never used, but nobody does that.
This simplifies the code a little and lets all methods be const.
Instead of using lots of small scattered reads to read the FST,
only one big read is used, which is more efficient.
This also means that the FST only allocates memory once and stores all
strings close to each other - good for the CPU cache. The file info
objects use pointers to this FST memory of containing data themselves.
Keeping around the big m_FileInfoVector containing objects with only
pointers is a bit unnecessary, but that will be fixed soon.
Instead of calling GetPathFromFSTOffset for every file info, FindFileInfo
now only looks at names in directories that are included in the path.
For the common case of searching for "opening.bnr", this means that
only root-level files and directories have to be searched through.
Some callers already have the file info, making the relatively slow
FindFileInfo calls unnecessary. Callers that didn't have the file info
will now need to call FindFileInfo on their own.
Some callers (i.e. ISOProperties) don't want the full path, so giving them
it is unnecessary. Those that do want it can use GetPathFromFSTOffset.
Not storing full paths everywhere also saves a small bit of RAM and is
necessary for a later commit. The code isn't especially pretty right now
(callers need to use FST offsets...) but it'll become better later.
GC/Wii filesystem internals shouldn't be exposed to other classes.
This change isn't especially useful by itself, but it opens up the
way for some neat stuff in the following commits.
You may want to read the PR #2047 comments before reading this.
Dolphin attempts to support an unencrypted type of Wii discs
that apparently is identified by a 4-byte integer at 0x60
being non-zero. I don't know what discs (if any) would be
using that format, so I haven't been able to test Dolphin's
support for it, but it has probably been broken for a while.
The old implementation is very short but also strange.
In CreateVolumeFromFilename, we read a 4-byte integer from
0x60, and if it's non-zero, we create a CVolumeGC object
instead of a CVolumeWiiCrypted object. This might seem like
it makes no sense, but it presumably worked in the past
because IsVolumeWiiDisc used to check the volume type by
reading the magic word for Wii straight from the disc,
meaning that CVolumeGC objects representing unencrypted Wii
discs would be treated as Wii discs by pretty much all of
Dolphin's code except for the volume implementation code.
(It wasn't possible to simply use CVolumeWiiCrypted, because
that class only handled encrypted discs, like the name says.)
However, that stopped working as intended because of ace0607.
And furthermore, bb93336 made it even more broken by making
parts of Dolphin expect that data read from Wii discs needed
to be decrypted (rather than the volume implementation
implicitly deciding whether to decrypt when Read was called).
Disclaimer: Like I said before, I haven't been able to test
any of this because I don't have any discs that use this
unencrypted Wii disc format, so this is all theoretical.
Later, PR #2047 tried to remove Dolphin's support for
the unencrypted Wii disc format because seemingly no
discs used it, but the PR got closed without being merged.
At the end of that PR, I said that I would make a new PR
with a better implementation for the format after PR #2353
was merged. Now that PR #2353 is merged (two years later...)
and PR #5521 is merged, the new implementation was easy to
make, and here it is!
Untested.
Normal users don't care about it. In fact, people care so
little about it that the Wii implementation of it was broken
starting from when it was implemented (eb65601) to 7 years
later (e0a47c1), apparently without anyone reporting it.
This makes the interface slightly cleaner and a bit more consistent
with the other getters. Still not fully the same, since the others
don't really handle failures with std::optional; but at least the
value is returned by value now, as opposed to having the function
take a pointer to a u64.
This commit moves the write function to where it should be (IOS),
especially when ES::ImportTicket() is the only place to use it.
Prevents misusing the ticket import function, and removes one unsafe
direct write to the NAND that does not go through IOS.
This also fixes the destination path: the session root is the one which
should be used for determining the ticket path, not the configured one.
The whole NANDContentLoader stuff is truly awful and will be removed
as soon as possible.
For now, this fixes a bug that was exposed by std::optional::operator*.
gcc complains that the printf %x formatting instruction expects an
'unsigned int' but we pass a 'size_t'. We add the 'z' length formatting
specifier used for 'size_t'
This file is pretty small now that it doesn't handle Wii
partitions anymore, so let's move its contents to Volume.cpp.
This is also more consistent with how blob creation works.
This happened to work without any problems because the only way for a
file system to be invalid was to not have the right GC/Wii magic word
in the unencrypted area, and a volume could not be created without
having the right GC/Wii magic word there. Now that file systems read
the magic word from a partition instead, a fix is needed.
I replaced m_OffsetShift with m_Wii in bb93336 to support
the decrypt parameter for read functions. Doing that is no
longer necessary, so m_offset_shift is now used like before.
By removing mutable state in VolumeWiiCrypted, this change makes
partition-related code simpler. It also gets rid of other ugly things,
like ISOProperties's "over 9000" loop that creates a list of
partitions by trying possible combinations, and DiscScrubber's
volume swapping that recreates the entire volume when it needs to
change partition.
Direct access to the WAD bytes is required to read contents with proper
padding data (since they can sometimes end up being outside of the
data app section). Allowing the whole buffer to be accessed directly
would be error prone, so this commit adds GetContent() to WiiWAD
for getting raw content data by index.
Just like DeleteTitle, Using CNANDContentManager is overkill,
inefficient and useless. And it results in a few failures in
situations where a delete should just always work.
But here it gets bonus points, because it manages to actually use
the TMD for deleting contents, when IOS does none of that and just
deletes files ending with .app in the title content directory. :)
Fixes warning:
```
Source/Core/DiscIO/NANDImporter.cpp:55:17: warning: format specifies type 'unsigned long' but the argument has type 'u64' (aka 'unsigned long long') [-Wformat]
file.GetSize(), NAND_BIN_SIZE);
^~~~~~~~~~~~~~
1 warning generated.
```
This will be required for permission checks in the future.
Note that this is only for the PPC as we do not have actual processes.
Keeping track of other modules' UIDs/GIDs is virtually useless anyway.
UID/GID changes are implemented in the following functions:
* ES_Launch
* ES_DIVerify
ES_SetUid is not implemented yet because it'd need further changes.
Fixes file ordering in games that use ASCII characters between lowercase
'z' and uppercase 'A' (underscores).
MySims Kingdom has the files "terrainLightMapTinted.shader",
"terrainLightMapTintedGrid.shader", and
"terrainLightMapTinted_no_shadow.shader". In lowercase,
"terrainLightMapTinted_no_shadow.shader" comes before
"terrainLightMapTinted.shader" and "terrainLightMapTintedGrid.shader",
which is invalid.
The size field in FSTEntry contains the total amount of children, not
including the parent, but the parent needs to be included.
VolumeDirectory: Fix off-by-one in entry count calculation
The size field in FSTEntry contains the total amount of children, not
including the parent, but the parent needs to be included.
Fixes warnings:
```
../Source/Core/DiscIO/VolumeGC.cpp: In member function 'virtual u8 DiscIO::CVolumeGC::GetDiscNumber() const':
../Source/Core/DiscIO/VolumeGC.cpp:178:10: error: 'disc_number' may be used uninitialized in this function [-Werror=maybe-uninitialized]
return disc_number;
^
../Source/Core/DiscIO/VolumeWiiCrypted.cpp: In member function 'virtual u8 DiscIO::CVolumeWiiCrypted::GetDiscNumber() const':
../Source/Core/DiscIO/VolumeWiiCrypted.cpp:258:10: error: 'disc_number' may be used uninitialized in this function [-Werror=maybe-uninitialized]
return disc_number;
^
../Source/Core/DiscIO/VolumeWiiCrypted.cpp: In member function 'virtual IOS::ES::TMDReader DiscIO::CVolumeWiiCrypted::GetTMD() const':
../Source/Core/DiscIO/VolumeWiiCrypted.cpp:123:20: error: 'tmd_address' may be used uninitialized in this function [-Werror=maybe-uninitialized]
tmd_address <<= 2;
^
```
Dolphin assumes that content 0 is opening.bnr, without checking
whether content 0 exists or if it is even supposed to be there (it's
only there for channels). This results in sometimes reading garbage.
This adds a check to only try to read names from content 0's header
if the title is a channel (channel, system channel or game channel).
The Tools > Load System Menu option displays the version of the
installed Wii Menu. This commit changes the way we display that
version, like so: "Load System Menu 514P" -> "Load System Menu 4.3E"
The numbers are from http://wiibrew.org/wiki/System_Menu
5.0-2712 made ES's code for setting the game ID use the
title ID converted to hex (except for disc titles) instead
of using a 6-char game ID like before. Then, 5.0-2830 made
us use that code even when loading game INIs. This breaks the
expectations of both users and the game INIs we ship with.
This commit makes Dolphin use 6-char game IDs for all
titles (unless the 6-char ID would contain unprintable
characters, which is the case with e.g. the Wii Menu).
I'm also putting unprintability checks in VolumeWad
for consistency.
Advantages:
* Simpler code in general
* No extra volume objects created
* Now actually notices if the disc or partition gets
changed while the core is running
* No longer picks up on disc access done by the GUI
(it used to do so as long as the core was running)
* Gets rid of a Core dependency in DiscIO
There are two performance disadvantages:
* FileMonitor is now a bit slower when used with VolumeDirectory
because FileMonitor now always uses the FileSystemGCWii code
for finding filenames instead of VolumeDirectory finding the
filename on its own and directly hooking into FileMonitor.
But this isn't such a big deal, because it's happening on the
DVD thread, and my currently unmerged file system PR will make
FileSystemGCWii's file finding code about as fast as
VolumeDirectory's.
* FileMonitor's creation of the file system object is now
done on the CPU thread instead of the DVD thread, and
it will be done even if FileMonitor logging is disabled.
This will be fixed in the next commit.
There's no point in creating a volume without a blob,
since essentially all the functionality of a volume
requires a blob to be used.
Also, VolumeCreator doesn't support creating volumes
without blobs (it can't even figure out the volume type
unless it gets a blob), so it's currently impossible
for a volume to be created without a blob.
Most of the Volume code was written before this
convenience function was added. Let's use it more.
Also deleting m_pReader nullptr checks that are
unnecessary because of Read (which ReadSwapped calls)
already having a nullptr check.
This moves all the byte swapping utilities into a header named Swap.h.
A dedicated header is much more preferable here due to the size of the
code itself. In general usage throughout the codebase, CommonFuncs.h was
generally only included for these functions anyway. These being in their
own header avoids dumping the lesser used utilities into scope. As well
as providing a localized area for more utilities related to byte
swapping in the future (should they be needed). This also makes it nicer
to identify which files depend on the byte swapping utilities in
particular.
Since this is a completely new header, moving the code uncovered a few
indirect includes, as well as making some other inclusions unnecessary.
This changes ES to keep track of the active title properly,
just like IOS:
* It is NOT changed on resource manager open/close.
* It is reset on IOS reload.
* It is changed by ES_DIVerify and ES_Launch.
IOS stores the active title in a structure like this:
struct ESTitleContext
{
Ticket* ticket;
TMD* tmd;
u32 active;
};
With this commit, we also do keep the Ticket and TMD around. This
makes some of the DI ioctlvs (which return data about the current
active title) trivial to implement in the future.
This fixes the System Menu not being able to see update partitions
and also allows us to change Dolphin's active game info in the future.
We already have a TMDReader, so let's actually use it.
And move ESFormats to IOS::ES, since it's definitely part of IOS.
This adds a DiscIO dependency on Core which will be fixed in a
follow-up PR.
This prevents Dolphin from writing to /sys/uid.sys (on the host; root
partition) when installing a WAD before starting emulation, because
the session root is not initialized at that moment.
Incidentally, this also gets rid of a singleton.
This fixes a regression from 5.0-1556, but I don't know why
the regression occurred or why this fixes it. (Many games
failed to fully boot - I tried Metroid Prime and Twilight
Princess (both GC), and they never got to the title screen.)
These two functions load either a signed ticket or a raw ticket from the
emulated NAND.
The ticket signature skip is refactored out of the ticket writing in
order to be usable by the raw ticket reading function.
Refactor the existing DiscIO::AddTicket to not require the caller to
pass the requested title ID. We do not have the title ID in the ES case,
and it needs to be extracted from the ticket. Since this is always a
safe operation to do (title ID is always in the ticket), the
implementation is made default.
Instead of needing different switch cases for
converting countries to regions in multiple places,
we now only need a single country-to-region switch case
(in DiscIO/Enums.cpp), and we get a nice Region type.
We can't rely on the OS returning files and directories
in a deterministic order, so we should sort them on our own
if we want VolumeDirectory to work for movies and netplay.
These disc images are encrypted and signed using a different set of keys.
We only care about the master key, so we check the signature issuer. If
it matches the debug issuer, then we'll use the RVT-R key. Otherwise,
the previous set of common and Korean keys are used.
This would previously fail to compile when included in files that do not
include FileUtil.h due to lack of a type declaration.
This moves the constructor and destructor into the cpp file in order to
satisfy the requirements of unique_ptr construction and deletion. That is,
unique_ptr requires a concrete type at the point of construction and
destruction. If the constructor or destructor is left in the header, then
at the point of construction or destruction, IOFile will still be
considered an incomplete type, as unique_ptr's deleter will still only be
able to see the forward declaration, which it can't use.
These functions don't actually depend on any state from the class
instance, so they don't really belong in the header, and are just
an implementation detail.
At first there weren't many enums in Volume.h, but the number has been
growing, and I'm planning to add one more for regions. To not make
Volume.h too large, and to avoid needing to include Volume.h in code
that doesn't use volume objects, I'm moving the enums to a new file.
I'm also turning them into enum classes while I'm at it.
If banner loading fails once, it will very likely fail again.
Setting m_banner_loaded to true when banner loading fails
prevents LoadBannerFile from wasting time if it's called again.
Banner loading requires loading the file system, which takes a
noticeable amount of time, so this matters.
DriveReader::m_size was never initialized which was indirectly
causing CGameListCtrl to crash Dolphin when it tried to insert a
character at a negative index in a string.
Reading one sector at a time is very inefficient and appears to
be causing timing issues during boot so SectorReader has been
enhanced to support batching.
SectorReader has been given a working cache system.
instead, leave all the management with the NANDContentLoader.
for file data (directly on the NAND), this opens the file on-demand and
returns the requested chunk when asked for it.
for on-the-fly decrypted WAD data, we just keep the decoded buffer in
memory, like we've done before - except that we don't give away any objects
we don't want to.
Using the XPM format for images has become a maintenance problem because
people don't know how to create them. This commit removes all XPM images
and all C files that contain PNG images. DolphinWX now uses the PNGs
in the Resources folder instead, just like DolphinQt and DolphinQt2 do.
Callers can now check whether reads fail, either by checking the return
value or by setting the buffer to a known bad value and seeing if it stays
untouched. I've added error checks to FileSystemGCWii and Boot_BS2Emu,
but not to Boot since it doesn't check any of its other reads either.
Rather than rely on the developer to do the right thing,
just make the default behavior safely deallocate resources.
If shared semantics are ever needed in the future, the
constructor that takes a unique_ptr for shared_ptr can
be used.
If a disc image is malformed in a specific way, Dolphin
will try to allocate a lot of memory, making it crash.
To avoid that, this change adds an artificial limit for
the size of file systems that Dolphin will try to load.
The offset in a directory FileInfo is an FST offset, not a disc offset.
Treating it as a disc offset doesn't make sense. Directories don't take
up space outside of the FST, so they don't need to be marked as used. The
old behavior may have prevented some scrubbable areas from being scrubbed.
It's used by both the GUI to do things like install WADs and check up on
the system menu, in which case the global root should be used, and by
/dev/es, in which case the local one should. The latter isn't
*terribly* useful today, since no contents will ever be installed in
temporary roots (although it's still relevant for data directories), but
converting the whole thing makes sense because then it will Just Work
once the entire NAND is synced.
Because it would have been a bit of work to split it up (but I can if
desired), this commit also contains some basic cleanup of
NANDContentLoader:
(1) The useless interface class INANDContentLoader is removed and the
methods are changed to just return CNANDContentLoader (the only
implementation);
(2) CNANDContentManager is changed to use unique_ptr and cleaned up a
bit.
04fcb72 fixed an issue with reading the Wii FST size, but I found a second
issue when working on PR #2820 - the size must be shifted left by 2.
DiscScrubber and Boot already do this correctly using separate code.
read_size remained 0 when the "Read beyond end of disc" error occurs,
which made the while (nbytes) loop never end. As a fix, SeekToCluster
now explicitly sets available to 0 when the error occurs, and Read
checks for it.
GC games with long names store two variations of the name in
opening.bnr. This makes the shorter of those names available.
For volumes other than GC discs, prefer_long is ignored.
ISOFile and GameFile were using IsWiiDisc() and IsWadFile() to set
an enum value. The volume might as well return an enum directly.
I increased the Qt CACHE_REVISION because m_platform now is saved as u32
instead of int, but increasing the wx CACHE_REVISION is not necessary.
I tried to change messages that contained instructions for users,
while avoiding messages that are so technical that most users
wouldn't understand them even if they were in the right language.
Eventually, netplay will be able to use the host's NAND, but this could
still be useful in some cases; for TAS it definitely makes sense to have
a way to avoid using any preexisting NAND.
In terms of implementation: remove D_WIIUSER_IDX, which was just WIIROOT
+ "/", as well as some other indices which are pointless to have as
separate variables rather than just using the actual path (fixed, since
they're actual Wii NAND paths) at the call site. Then split off
D_SESSION_WIIROOT_IDX, which can point to the dummy NAND directory, from
D_WIIROOT_IDX, which always points to the "real" one the user
configured.
- FileSearch is now just one function, and it converts the original glob
into a regex on all platforms rather than relying on native Windows
pattern matching on there and a complete hack elsewhere. It now
supports recursion out of the box rather than manually expanding
into a full list of directories in multiple call sites.
- This adds a GCC >= 4.9 dependency due to older versions having
outright broken <regex>. MSVC is fine with it.
- ScanDirectoryTree returns the parent entry rather than filling parts
of it in via reference. The count is now stored in the entry like it
was for subdirectories.
- .glsl file search is now done with DoFileSearch.
- IOCTLV_READ_DIR now uses ScanDirectoryTree directly and sorts the
results after replacements for better determinism.
This is written so that the result of GetCompanyFromID never is cached
(except on Android?). Caching is unnecessary because the string can be
obtained quickly at runtime, and not caching it means that the cache
doesn't have to be invalidated when GetCompanyFromID is edited.
This is intended to better separate it from GetNames and to clarify
that this name originally wasn't meant to be shown to users.
The ISOProperties GUI is also updated, mainly because labeling
the long banner name "short name" was confusing.
Having some data available in banner loaders and some other data
data available in volumes gets messy, especially with GetNames(),
which is available in both but returns different results
depending on which one is used. This change drops support
for reading names and descriptions from Wii save data.
JosJuice