By calling ZSTD_CCtx_setPledgedSrcSize, we can let zstd know
how large a chunk is going to be before which start compressing
it, which lets zstd avoid allocating more memory than needed
for various internal buffers. This greatly reduces the RAM usage
when using a high compression level with a small chunk size,
and doesn't have much of an effect in other circumstances.
A side effect of calling ZSTD_CCtx_setPledgedSrcSize is that
zstd by default will write the uncompressed size into the
compressed data stream as metadata. In order to save space,
and since the decompressed size can be figured out through
the structure of the RVZ format anyway, we disable writing
the uncompressed size by setting ZSTD_c_contentSizeFlag to 0.
It's possible (but rare) for a WIA or RVZ file to support
this for some partitions but not all, and for the game and
the blob code to disagree on how large a partition is.
The heuristic was not allocating enough space for Metroid: Other M,
at least when using the default settings. (This didn't break the
file, it just caused some headers to be placed at the end of the
file instead of at the start and wasted a few hundred kilobytes.)
The new hash check catches essentially all desync problems
that VolumeVerifier can catch, so from the user's perspective,
such problems will result in Dolphin refusing to start the
game on netplay rather than actually getting a desync.
The "FindLibLZMA.cmake" module in CMake versions prior to 3.14 do not
set an alias like how Externals/liblzma/CMakeLists.txt does, so builds
performed using one of those older CMake versions will fail if the
system LZMA library is detected. To fix this, we need to link against
"lzma" instead of "LibLZMA::LibLZMA".
Fixes: b59ef81a7e ("WIA: Implement bzip2, LZMA, and LZMA2 decompression")
When I first made VolumeVerifier, I figured that the distinction
between an unsigned ticket and an unsigned TMD was a technical
detail that users would have no reason to care about. However,
while this might be true for discs, it isn't equally true for
WADs, due to the widespread practice of fakesigning tickets to
set the console ID to 0. This practice does not require
fakesigning the TMD (though apparently people do it anyway,
at least sometimes...), and the presence of a correctly signed
TMD is a useful indicator that the contents have not been
tampered with, even if the ticket isn't correctly signed.
Instead of comparing the game ID, revision, disc number and name,
we can compare a hash of important parts of the disc including
all the aforementioned data but also additional data such as the
FST. The primary reason why I'm making this change is to let us
catch more desyncs before they happen, but this should also fix
https://bugs.dolphin-emu.org/issues/12115. As a bonus, the UI can
now distinguish the case where a client doesn't have the game at
all from the case where a client has the wrong version of the game.
It is my opinion that nobody should use NKit disc images without
being aware of the drawbacks of them. Since it seems like almost
nobody who is using NKit disc images knows what NKit is (hmm, now
how could that have happened...?), I am adding a warning to Dolphin
so that you can't run NKit disc images without finding out about the
drawbacks. In case someone really does want to use NKit disc images,
the warning has a "Don't show this again" option. Unfortunately, I
can't retroactively add the warning where it's most needed:
in Dolphin 5.0, which does not support Wii NKit disc images.
This could cause read errors if chunks were laid out a certain
way in the file and the whole chunk wasn't being read at once.
Should fix https://bugs.dolphin-emu.org/issues/12184.
PURGE isn't especially useful, while requiring some annoying
special handling in the file format. If you want no compression,
use NONE. If you want fast compression, use Zstandard.
Gets rid of the need to seek to the end of the file
when opening a file.
The downside of this is that we waste a little space,
since we can't know in advance exactly how much
space the compressed parts of the headers will need.
This is useful for the way Dolphin scrubs Wii discs.
The encrypted data is what gets zeroed out, but this
zeroed out data then gets decrypted before being stored,
and the resulting data does not compress well.
However, each block of decrypted scrubbed data is
identical given the same encryption key, and there's
nothing stopping us from making multiple group entries
point to the same offset in the file, so we only have
to store one copy of this data per partition.
For reference, wit zeroes out the decrypted data,
but Dolphin's WIA writer can't do this because it currently
doesn't know which parts of the disc are scrubbed.
This is also useful for things such as storing Datel discs
full of 0x55 blocks (repesenting unreadable blocks)
without compression enabled.
Fixes https://bugs.dolphin-emu.org/issues/10654.
To quote the documenation file included with the program tgctogcm:
"TGC's are miniaturized .gcm images with a 32kB header.
The embedded gcm contains some bogus data, namely:
-FST Location (0x424 in gcm)
-DOL Location (0x420 in gcm)
-FST File offsets (all files are offset/spoofed by a certain amount)"
Dolphin has been handling the values at 0x420 and 0x424 by simply
overwriting them with a working value (just like tgctogcm does),
but it has used a different approach for the file offsets in the FST.
Instead of changing the offsets that are stored in the FST, Dolphin
changed where the files actually are placed on the virtual disc.
My hope was that this would make the loading times more accurate to
how they are when running a TGC file as part of a larger disc.
However, there are TGC files where we would need to move files
backwards on the disc in order to do this (this is what issue
10654 is about), so the approach we have been using is flawed.
This change makes Dolphin overwrite offsets in the FST instead, like
tgctogcm does. Other than making Dolphin handle the affected TGC files
correctly, this change also makes it so that unnecessary padding data
isn't written if you use Dolphin to convert a TGC file to an ISO file.
This feature is not actually implemented in Dolphin as of now, but I'm
planning to add it in the near future as part of a larger feature.
This is intended to catch WIA files which have been created using
wit's default parameters (40 MiB block size), once the WIA PR is
merged. The check does however also work for GCZ files – not that
I think anyone has a GCZ file with a block size that large.
The code was actually already rather well adapted for this.
We more or less just have to skip ParseDisc and run
ParsePartitionData directly. This required the PartitionHeader
struct to be removed (which wasn't that useful anyway).
If we start 31 KiB into a 32 KiB block and want to mark 2 KiB
of data as used, we need to mark 2 blocks as used, not just 1.
This problem is avoided when calling MarkAsUsed from
MarkAsUsedE, since MarkAsUsedE aligns to 32 KiB on its own.
Most calls to MarkAsUsed are from MarkAsUsedE, which is why
this hasn't been a noticeable problem in the past.
The constant DESIRED_BUFFER_SIZE was determined by multiplying the
old hardcoded value 32 with the default GCZ block size 16 KiB.
Not sure if it actually is the best value, but it seems fine.
Add a function that safely returns whether a character is printable
i.e. whether 0x20 <= c <= 0x7e is true.
This is done in several places in our codebase and it's easy to run
into undefined behaviour if the C version defined in <cctype>
is used instead of this one, since its behaviour is undefined
if the character is not representable as an unsigned char.
This fixes MemoryViewWidget.
A small, nonexhaustive set of warning fixes. The DiscIO Volume change
is a workaround for a GCC bug [1] that causes returning an unengaged
std::optional to emit annoying -Wmaybe-uninitialized warnings.
This last change alone fixes pages upon pages of warnings since
Volume.h is included from several files.
-Wstringop-truncation is another irrelevant warning for us, but
unfortunately there seems to be no way to disable it without
adding ugly pragmas wherever the warning appears.
string_view is a thin wrapper around C strings, so it's more efficient
for constant strings than C++ strings.
The unordered_set<> also adds extra runtime overhead. For small arrays,
a simple linear search works. For larger arrays, std::binary_search()
works better than linear but without the unordered_set<> overhead.
ShouldBeDualLayer(): Removed a duplicate "SK8X52" entry.
Fixes using DirectoryBlob on extracted games that were unencrypted
prior to being extracted.
(One day I'll make DirectoryBlob actually support raw reads and then
the order of these two won't matter...)
This happens if someone manually downloads a regular datfile from
redump.org and puts it where Dolphin stores datfiles. Dolphin needs
"special" datfiles that contain fields for serials and versions.
Before this change, all discs (except Datel discs) would show up as
"Unknown disc" when using a regular datfile.
This was making it impossible to use the Redump.org integration
without first manually creating a Redump folder in the Cache folder.
https://bugs.dolphin-emu.org/issues/11885
PARTITION_NONE technically has a runtime static constructor otherwise.
This allows compile-time instances of Partition to be created without
the use of a static constructor.
The old implementation of this was not able to distinguish between
a title that had the common key index set to 1 because it actually
was Korean and a title that had the common key index set to 1 due to
fakesigning. This new implementation solves the problem by
decrypting a content with each possible common key and checking
which result matches the provided SHA-1 hash.
The problem that the old implementation causes has only been reported
to affect a certain pirated WAD of Chronos Twins DX (WC6EUP), but it's
possible that the problem would start affecting more WADs if we add
support for the vWii common key (which uses index 2). Adding support
for the vWii common key would also prevent us from using the simpler
solution of always forcing the index to 0 if the title is not Korean.
...in addition to the existing function CreateVolume
(renamed from CreateVolumeFromFilename).
Lets code easily add constraints such as not letting the user
select a WAD file when using the disc changing functionality.
Since C++17, non-member std::size() is present in the standard library
which also operates on regular C arrays. Given that, we can just replace
usages of ArraySize with that where applicable.
In many cases, we can just change the actual C array ArraySize() was
called on into a std::array and just use its .size() member function
instead.
In some other cases, we can collapse the loops they were used in, into a
ranged-for loop, eliminating the need for en explicit bounds query.
Removes redundant initializers from the constructor and provides
initializers for all members that don't already have one for consistency
(and deterministic initial state).
Given the volume verifier has quite a few non-trivial object within it,
it's best to default the destructor within the cpp file to prevent
inlining complex destruction logic elsewhere, while also making it nicer
if a forward-declared type is ever used in a member variable.
The difference between Dolphin's game IDs and GameTDB's game IDs
is that GameTDB uses four characters for non-disc titles, whereas
Dolphin uses six characters for all titles.
This fixes:
- TitleDatabase considering Datel discs to be NHL Hitz 2002
- Gecko code downloading not working for discs with IDs starting with P
- Cover downloading mixing up discs with channels (e.g. Mario Kart Wii
and Mario Kart Channel) and making extra HTTP requests. (Android was
actually doing a better job at this than DolphinQt!)
These country codes have the unfortunate property that they are used
by Wii disc games in two different regions. We already correct for this
in VolumeGC::GetCountry and VolumeWii::GetCountry, so this commit
shouldn't really have any effect on how the game list behaves,
but it will be useful if we in the future would want to call
CountrySwitch directly without having extra code in the caller for
handling region weirdness.
The header of a Wii disc can be read from two places: The
unencrypted area at the beginning of the disc, or the beginning of
the game partition. The two copies are usually identical (except
for 0x60 and 0x61), but there are exceptions. For most of Dolphin's
history, we have been reading from the header inside the game
partition when getting metadata. This was however not the case
starting with 4.0-4901 and ending with 5.0-3762. This commit once
again makes Dolphin read metadata from the unencrypted header,
because of the following reasons that I recently was informed about:
- The "pink fish" disc has the game ID 410E01 in the unencrypted
header but the placeholder game ID RELSAB in the partition header.
- The revisions of some games differ between the two headers,
with the unencrypted one making more sense.
(See https://bugs.dolphin-emu.org/issues/11387)
For better or worse, this also means that sloppily hacked games where
only the game ID in the unencrypted header has been changed now will
use that modified game ID. And unlike with the partition header,
there is no signing or hashing that can tell us whether the
unencrypted header has been modified by someone other than Nintendo.
ChunkFile doesn't use any of the file utilities, so we can drop these
headers to avoid pulling in unnecessary dependencies. This also
uncovered a few indirect inclusions.
We can just use std::any_of here to collapse the checking code down to a
single assignment as opposed to a loop. This also slightly improves on
the existing code, as this won't continue to iterate through the cluster
metadata if an entry that's non-zero is encountered.
These disc images are only used on dev units and not retail units.
There are two important differences compared to normal Wii disc images:
- The data starts 0x8000 bytes into each partition instead of 0x20000
- The data of a partition is stored unencrypted and contains no hashes
Our old implementation was just guesswork and doesn't work at all.
According to testing by GerbilSoft, this commit's implementation
is able to read and extract files in the filesystem correctly,
but the tested game still isn't able to boot. (It's thanks to their
info about unencrypted disc images that I was able to make this commit.)
Since all FS access will go through the new FS interface (PR #6421)
in order to keep track of metadata properly, there is no need to return
absolute paths anymore.
In fact, returning host paths is a roadblock to using the FS interface.
This starts the migration work by adding a way to get paths that are
relative to the Wii NAND instead of always getting absolute paths
on the host FS.
To prepare for future changes, this commit also makes returned paths
canonical by removing the trailing slash when it's unneeded.
Eventually, once everything has been migrated to the new interface,
we can remove the "from" parameter.
The original reason I wanted to do this was so that we can replace
the Android-specific code with this in the future, but of course,
just deduplicating between DolphinWX and DolphinQt2 is nice too.
Fixes:
- DolphinQt2 showing the wrong size for split WBFS disc images.
- DolphinQt2 being case sensitive when checking if a file is a DOL/ELF.
- DolphinQt2 not detecting when a Wii banner has become available
after the game list cache was created.
Removes:
- DolphinWX's ability to load PNGs as custom banners. But it was
already rather broken (see https://bugs.dolphin-emu.org/issues/10365
and https://bugs.dolphin-emu.org/issues/10366). The reason I removed
this was because PNG decoding relied on wx code and we don't have any
good non-wx/Qt code for loading PNG files right now (let's not use
SOIL), but we should be able to use libpng directly to implement PNG
loading in the future.
- DolphinQt2's ability to ignore a cached game if the last modified
time differs. We currently don't have a non-wx/Qt way to get the time.
We shouldn't try to create folder names that contain characters
such as : or / since they are forbidden or have special meanings.
(No officially released disc uses such characters, though.)
Use std::string(cstring, strnlen(cstring, max_length)) instead of
trying to remove extra null characters manually, which is a bit
ugly and error prone.
And indeed, the original code contained a bug which would cause
extra NULLs to not be removed at all if the string did not
end with a NULL -- causing issues down the road when constructing
paths for sub-entries.
This removes the hack that enables directly booting from WADs
without installing them first for the following reasons:
1. It makes the NAND content handling much more complicated than what
it should be and makes future changes like permissions or booting
NAND titles without a WAD more annoying to implement.
Because of this hack, we needed an extra level of abstraction
(NANDContent*) which has to read tons of things from the NAND, even
most of the time it's useless. This in turn forces us to have
caching, which is known to break titles and requires manual cache
invalidations. Annoying and error prone.
2. It prevents the WAD boot code from being easily accurate. With this
change, we can simply reuse the existing launch code, and ask IOS
to launch the title from the NAND.
3. The hack did not work that well since it did not cover a lot of ES
commands. And it works even less since the ES accuracy fixes.
This results in Dolphin returning inconsistent results: a
lot of the ES "DI" commands will just fail because the active title
is not installed on the NAND. uid.sys is not changed, etc.
And I'm not even talking about FS stuff -- where this would still
totally fail, unless we add even more unnecessary hacks.
This is not just theoretical -- the system menu and the Wii Shop are
known to behave strangely because the hack damages the NAND
structure, and we've already had several users report issues.
This commit makes it so WADs are always installed prior to launching.
A future commit will remove any code that was there only for the hack.
This updates the maker data to (mostly) mirror that of the Wiki:
https://wiki.dolphin-emu.org/index.php?title=GameIDs
Only maker ids from that page are now included in Dolphin. This
means no homebrew/unofficial makers.
Also, separate multiple maker names with a slash
Some of my WiiWare games does not have a maker :
- Blue's Journey : EAFPJ8
- Magician Lord : EACPJ8
- The King of Fighters '94 : EAGPJ8
- The Last Ninja : C9XPGX
- World Games : C9ZPGX
I noticed the Strong Bad games, FAST - Racing League, and Tetris Party
were lacking info in the game lists' maker column.
This adds the information based on the games' MakerID.
This adds a WiiWad constructor that takes a BlobReader, so that the
class can be used with more than just files from the host filesystem.
Required for using WiiWad with WADs from update partitions.
There were two problems with this:
1. If the starting offset was beyond the end of the disc,
we would dereference an invalid iterator.
2. The data beyond the end of the disc was non-deterministic.
Having DiscContents with size 0 would mean that some DiscContents
might not get added to the std::set because of them comparing
identically to another DiscContent.
This replaces an older piece of code in WriteDirectory that ensures
that no two files have the same starting offset. (We now care about
the ending offset, not the starting offset. The new solution both
ensures that no two files have the same ending offset and that no
two files have the same starting offset.)
JosJuice