Commit Graph

1151 Commits

Author SHA1 Message Date
JosJuice e3d291a529 WIA: Check the internal WIA hashes 2020-06-17 13:43:53 +02:00
JosJuice 827437c036 WIA: Fix the handling of chunk sizes larger than 2 MiB 2020-06-17 13:43:53 +02:00
JosJuice 0b407228b7 WIA: Add documentation 2020-06-17 13:43:53 +02:00
JosJuice 01a77ae8a1 WIA: Implement caching and partial decompression 2020-06-17 13:43:53 +02:00
JosJuice b59ef81a7e WIA: Implement bzip2, LZMA, and LZMA2 decompression 2020-06-17 13:43:52 +02:00
JosJuice 1579e061a3 WIA: Correctly handle data with size not divisible by chunk size 2020-06-17 13:43:52 +02:00
JosJuice 3c373c8aa8 WIA: Treat groups with size 0 as containing only zeroes 2020-06-17 13:43:52 +02:00
JosJuice 36991e2dde WIA: Implement PURGE decompression 2020-06-17 13:43:52 +02:00
JosJuice 3672bd79f3 WIA: Implement ReadWiiDecrypted 2020-06-17 13:43:52 +02:00
JosJuice 2a5fcc9c25 WIA: Add reading raw data 2020-06-17 13:43:51 +02:00
JosJuice 8da5d0c4fe Add an early version of WIABlobReader
It can currently only read the first 0x80 bytes of a disc image,
which is enough for identifying it but not for doing anything else.
2020-06-17 13:43:51 +02:00
JosJuice 75d032161f DiscIO: Rework the implementation of TGC reading
Fixes https://bugs.dolphin-emu.org/issues/10654.

To quote the documenation file included with the program tgctogcm:

"TGC's are miniaturized .gcm images with a 32kB header.
The embedded gcm contains some bogus data, namely:
-FST Location (0x424 in gcm)
-DOL Location (0x420 in gcm)
-FST File offsets (all files are offset/spoofed by a certain amount)"

Dolphin has been handling the values at 0x420 and 0x424 by simply
overwriting them with a working value (just like tgctogcm does),
but it has used a different approach for the file offsets in the FST.
Instead of changing the offsets that are stored in the FST, Dolphin
changed where the files actually are placed on the virtual disc.
My hope was that this would make the loading times more accurate to
how they are when running a TGC file as part of a larger disc.
However, there are TGC files where we would need to move files
backwards on the disc in order to do this (this is what issue
10654 is about), so the approach we have been using is flawed.

This change makes Dolphin overwrite offsets in the FST instead, like
tgctogcm does. Other than making Dolphin handle the affected TGC files
correctly, this change also makes it so that unnecessary padding data
isn't written if you use Dolphin to convert a TGC file to an ISO file.
This feature is not actually implemented in Dolphin as of now, but I'm
planning to add it in the near future as part of a larger feature.
2020-06-17 12:32:39 +02:00
JosJuice 162e3be82b Show an OSD message when running a disc image with a large block size
This is intended to catch WIA files which have been created using
wit's default parameters (40 MiB block size), once the WIA PR is
merged. The check does however also work for GCZ files – not that
I think anyone has a GCZ file with a block size that large.
2020-06-07 14:11:00 +02:00
JosJuice b93bbbf2e5 DiscIO: Implement multithreaded compression 2020-05-06 14:18:51 +02:00
degasus f8f592c345 Externals: Update minizip search path. 2020-04-29 12:41:36 +02:00
JosJuice 5b722b775a VolumeVerifier: Treat IOS9 as always present
Fixes a false positive for Mario Party 8.
2020-04-28 23:40:01 +02:00
JMC47 116cef572b
Merge pull request #8738 from JosJuice/convert-dialog
Replace the compress/uncompress actions with a convert dialog
2020-04-28 12:37:17 -04:00
JosJuice bacf0d629d VolumeVerifier: Show desync warning for dual layer discs too 2020-04-25 19:47:08 +02:00
JosJuice 6ffcbcee70 DiscIO: Move scrubbing code out of ConvertToGCZ
This way, scrubbing can also be performed when converting
to other formats.
2020-04-24 15:11:20 +02:00
JosJuice 04c7892b93 DiscIO: Add GameCube disc scrubbing support
The code was actually already rather well adapted for this.
We more or less just have to skip ParseDisc and run
ParsePartitionData directly. This required the PartitionHeader
struct to be removed (which wasn't that useful anyway).
2020-04-24 15:10:36 +02:00
JosJuice cefc2a7baa DiscIO: Fix edge case where blocks could get scrubbed accidentally
If we start 31 KiB into a 32 KiB block and want to mark 2 KiB
of data as used, we need to mark 2 blocks as used, not just 1.

This problem is avoided when calling MarkAsUsed from
MarkAsUsedE, since MarkAsUsedE aligns to 32 KiB on its own.
Most calls to MarkAsUsed are from MarkAsUsedE, which is why
this hasn't been a noticeable problem in the past.
2020-04-24 15:10:36 +02:00
JosJuice 42f6913bcc Move DiscIO::ConvertToPlain to FileBlob.cpp
There is no longer anything GCZ specific about it.
2020-04-24 15:10:35 +02:00
JosJuice 8a9597e32e DiscIO: Allow converting from formats other than ISO and GCZ
The constant DESIRED_BUFFER_SIZE was determined by multiplying the
old hardcoded value 32 with the default GCZ block size 16 KiB.
Not sure if it actually is the best value, but it seems fine.
2020-04-24 15:10:35 +02:00
JosJuice 432f342bc8 DiscIO: Use a struct for Wii hashes 2020-04-24 14:44:29 +02:00
JosJuice da9e0fb598 DiscIO: Parallelize the re-encryption code 2020-04-24 14:44:26 +02:00
JosJuice 319c508978 DiscIO: Implement re-encryption of Wii partition data 2020-04-24 14:24:12 +02:00
JosJuice a4c7100bcc DiscIO: Use partition data offset for ReadWiiDecrypted parameter
Instead of the partition offset (which is usually 0x20000 less).
2020-04-24 14:16:55 +02:00
JosJuice 19e9a9c945 DiscIO: Clean up decompression size calculation
We can use subtraction and std::min instead of
modulo and explicit if statements.

This commit does not change the behavior.
2020-04-15 22:15:40 +02:00
JosJuice 3aa463cdae DiscIO: Fix decompressing writing too much sometimes
This issue cannot happen with good dumps due to their size,
but it can happen with trimmed dumps.
2020-04-12 21:47:10 +02:00
JosJuice 26b21e3186 DiscIO: Fix decompressing writing too little sometimes
This issue cannot happen with good dumps due to their size,
but it can happen with trimmed dumps.
2020-04-12 21:45:55 +02:00
Jun Su d44c51b30c DiscIO: cleanup warnings of -Wmaybe-uninitialized
replace {} with std::nullopt.
2020-03-23 14:14:00 +08:00
Tilka 103b3abf66 Merge pull request #8621 from JosJuice/volumeverifier-invalid-partition
Fix VolumeVerifier not showing a problem for invalid partitions
2020-02-10 09:33:29 +00:00
JosJuice 829f3cff6f VolumeVerifier: Ignore invalid partitions in GetBiggestReferencedOffset
Otherwise GetBiggestReferencedOffset might treat garbage data as a valid
large offset, making Dolphin incorrectly say that the disc is too small.
2020-02-09 19:05:44 +01:00
JosJuice 07df6597a5 Fix VolumeVerifier not showing a problem for invalid partitions 2020-02-09 18:58:15 +01:00
Léo Lam 166e86b97f
Merge pull request #8619 from JosJuice/directoryblob-check-seek
DiscIO: Check for DirectoryBlob seek failure
2020-02-09 15:49:37 +01:00
JosJuice 9a348ae654 DiscIO: Check for DirectoryBlob seek failure
Someone reported a problem with DirectoryBlob at
https://forums.dolphin-emu.org/Thread-feature-request-thread?pid=502820#pid502820.
I'm not sure if the change in this commit actually fixes that problem,
but it's something I found that should be changed regardless.
2020-02-09 11:21:10 +01:00
JosJuice f8355d0f82 VolumeVerifier: Don't return early when file is too small
Not sure why that was there.
2020-02-02 17:09:58 +01:00
JosJuice e449d23929 VolumeVerifier: Don't show an assert for files that are too small 2020-02-02 17:09:58 +01:00
JosJuice de26fec0af VolumeVerifier: Report read errors to the user 2020-01-25 20:21:12 +01:00
Pierre Bourdon ea9b96370d
Merge pull request #8573 from JosJuice/ciso-size
DiscIO: Fix CISOFileReader::GetDataSize()
2020-01-22 19:20:36 +01:00
JosJuice 956c63ef9b DiscIO: Fix CISOFileReader::GetDataSize()
Fixes being unable to run CISO games after the merge of PR 8558.
2020-01-22 18:57:50 +01:00
Léo Lam 89b0ab2d22 StringUtil: Add IsPrintableCharacter and use it
Add a function that safely returns whether a character is printable
i.e. whether 0x20 <= c <= 0x7e is true.

This is done in several places in our codebase and it's easy to run
into undefined behaviour if the C version defined in <cctype>
is used instead of this one, since its behaviour is undefined
if the character is not representable as an unsigned char.

This fixes MemoryViewWidget.
2020-01-16 00:22:26 +01:00
JosJuice 297b790e4f DiscIO: Add out of bounds checks for blob reading 2020-01-14 18:59:31 +01:00
Léo Lam ad75215bb0 Fix several warnings
A small, nonexhaustive set of warning fixes. The DiscIO Volume change
is a workaround for a GCC bug [1] that causes returning an unengaged
std::optional to emit annoying -Wmaybe-uninitialized warnings.
This last change alone fixes pages upon pages of warnings since
Volume.h is included from several files.

-Wstringop-truncation is another irrelevant warning for us, but
unfortunately there seems to be no way to disable it without
adding ugly pragmas wherever the warning appears.
2020-01-04 12:11:39 +01:00
David Korth 1d7f128693 GetSysMenuVersionString(): Use a char instead of std::string for the region letter.
Append the region letter after determining the version number.
2019-12-29 23:45:02 -05:00
David Korth afe2e7de0f VolumeVerifier.cpp: Verify that the string arrays are sorted before using std::binary_sort().
Debug builds only, since this is wrapped in assert().
2019-12-29 23:42:55 -05:00
David Korth d660aba20f VolumeVerifier.cpp: Use arrays of string_view objects instead of strings.
string_view is a thin wrapper around C strings, so it's more efficient
for constant strings than C++ strings.

The unordered_set<> also adds extra runtime overhead. For small arrays,
a simple linear search works. For larger arrays, std::binary_search()
works better than linear but without the unordered_set<> overhead.

ShouldBeDualLayer(): Removed a duplicate "SK8X52" entry.
2019-12-29 23:42:55 -05:00
Stenzek 6fcb1c6c46 Add an ARM64 target to Visual Studio projects 2019-12-28 19:20:41 +10:00
JosJuice 711b36075c Fix scrubbing unencrypted Wii disc images (for real this time)
Unlike what my past self apparently thought, % is in fact not the
"round down" operator.

https://bugs.dolphin-emu.org/issues/11906
2019-12-08 19:41:40 +01:00
Stenzek dd23a1ee79 Update VS projects/solutions to VS2019 2019-11-30 13:42:52 +10:00
Anthony 44f85bbc5e
Merge pull request #8496 from JosJuice/volumewii-check-decrypted-order
VolumeWii: Check SupportsReadWiiDecrypted before m_encrypted
2019-11-27 15:33:43 -08:00
JosJuice 59633f5309 VolumeVerifier: Detect broken Super Paper Mario
https://bugs.dolphin-emu.org/issues/11900
2019-11-27 18:26:22 +01:00
JosJuice 23fc5f9e35 VolumeWii: Check SupportsReadWiiDecrypted before m_encrypted
Fixes using DirectoryBlob on extracted games that were unencrypted
prior to being extracted.

(One day I'll make DirectoryBlob actually support raw reads and then
the order of these two won't matter...)
2019-11-26 20:09:40 +01:00
JosJuice 4e713238d9 VolumeVerifier: Fix a copypaste error
This was completely wrong, but I guess it happened to work correctly
anyway due to timing reasons (MD5 is faster than SHA-1 + AES).
2019-11-22 13:39:04 +01:00
Connor McLaughlin 9700b1e320
Merge pull request #8367 from JosJuice/missing-company-ids
DiscIO: Add missing company IDs
2019-11-08 11:15:27 +10:00
JosJuice 56d37d773b RedumpVerifier: Show an error when datfile lacks serials or versions
This happens if someone manually downloads a regular datfile from
redump.org and puts it where Dolphin stores datfiles. Dolphin needs
"special" datfiles that contain fields for serials and versions.

Before this change, all discs (except Datel discs) would show up as
"Unknown disc" when using a regular datfile.
2019-11-02 16:49:20 +01:00
Mat M 78fad0aafa
Merge pull request #8443 from JosJuice/redumpverifier-datel-wii
RedumpVerifier: Fix handling of Datel Wii disc serials
2019-10-30 05:37:08 -04:00
JosJuice f9705fd117 Return nothing from VolumeWii::GetGameTDBID if Datel
RedumpVerifier relies on this.
2019-10-30 10:23:33 +01:00
JosJuice 42ec861469 RedumpVerifier: Fix handling of Datel Wii disc serials
GC Datel discs have empty serials, but Wii Datel discs have
serials starting with DS followed by some digits.
2019-10-30 10:11:24 +01:00
JosJuice 49977446dd RedumpVerifier: Don't crash on missing hyphen in serial 2019-10-30 09:58:37 +01:00
JosJuice e2ef248260 VolumeVerifier: Fix "The data partition is missing" false positive
My bad...
2019-10-30 09:38:18 +01:00
Mat M 427d4501bc
Merge pull request #8429 from JosJuice/redump-create-path
Create necessary folder when using Redump.org integration
2019-10-26 15:51:25 -04:00
JosJuice 2f1b2c6adf Create necessary folder when using Redump.org integration
This was making it impossible to use the Redump.org integration
without first manually creating a Redump folder in the Cache folder.
https://bugs.dolphin-emu.org/issues/11885
2019-10-25 11:07:50 +02:00
JosJuice 742aa765c6 VolumeVerifier: Don't show problems with good dumps of Wii Freeloaders 2019-10-25 10:03:12 +02:00
Mat M 780d34dbca
Merge pull request #8426 from JosJuice/volumeverifier-region-mismatch
VolumeVerifier: Improve region/game ID mismatch checking
2019-10-23 20:15:31 -04:00
JosJuice 2c79c63608
Merge pull request #8330 from JosJuice/redump-integration
VolumeVerifier: Add Redump.org integration
2019-10-23 16:43:52 +02:00
JosJuice 77b9a70d6c VolumeVerifier: Don't consider region mismatch for placeholder game IDs 2019-10-23 10:43:34 +02:00
JosJuice 97c4e8ff3a VolumeVerifier: Include revision when checking region mismatch 2019-10-23 10:39:19 +02:00
JosJuice f8a4975274 DiscIO: Make company names match GameTDB's list better 2019-09-25 13:38:39 +02:00
JosJuice 2300ce385e DiscIO: Add missing company IDs 2019-09-22 23:12:31 +02:00
Dexter Gerig 5c4b591536 Add Wii Mini system menu version detection 2019-09-21 12:55:49 -05:00
JosJuice 87c5e0b30e VolumeVerifier: Add Redump.org downloading 2019-08-25 12:53:19 +02:00
JosJuice 3eb360b818 VolumeVerifier: Add zip support for datfile 2019-08-24 16:37:19 +02:00
JosJuice 22933d8502 VolumeVerifier: Add datfile parsing 2019-08-24 16:37:19 +02:00
JosJuice 393709a45a Also override the SYSCONF country setting
Fixes https://bugs.dolphin-emu.org/issues/10066
2019-08-09 17:33:05 +02:00
JosJuice 8c8bab3657 VolumeVerifier: Show underdump warnings for WBFS/CISO too 2019-08-09 16:13:50 +02:00
Connor McLaughlin e87a3f91a9
Merge pull request #8303 from JosJuice/volumeverifier-very-small
Fix problems with running VolumeVerifier on very small disc images
2019-08-09 22:16:33 +10:00
JosJuice 107291dc5a Fix problems with running VolumeVerifier on very small disc images 2019-08-07 19:48:10 +02:00
JosJuice 34fb608dd6 VolumeVerifier: Multithreading 2019-08-06 16:42:59 +02:00
JosJuice f754a1a548 VolumeVerifier: Don't read data multiple times 2019-08-06 16:32:01 +02:00
Lioncash db3b31c246 DiscIO/Volume: Make Partition's interface constexpr
PARTITION_NONE technically has a runtime static constructor otherwise.
This allows compile-time instances of Partition to be created without
the use of a static constructor.
2019-08-02 18:25:09 -04:00
Léo Lam c8c1a0d747
Merge pull request #8253 from JosJuice/fakesigned-common-key-fix
Re-implement FixCommonKeyIndex for WAD files
2019-07-23 09:41:46 +08:00
Anthony 136264d340
Merge pull request #8245 from JosJuice/volumeverifier-wii-menu-region
Fix VolumeVerifier considering Wii Menu WADs to have wrong region
2019-07-22 15:06:07 -07:00
JosJuice 4b73d18eaa Re-implement FixCommonKeyIndex for WAD files
The old implementation of this was not able to distinguish between
a title that had the common key index set to 1 because it actually
was Korean and a title that had the common key index set to 1 due to
fakesigning. This new implementation solves the problem by
decrypting a content with each possible common key and checking
which result matches the provided SHA-1 hash.

The problem that the old implementation causes has only been reported
to affect a certain pirated WAD of Chronos Twins DX (WC6EUP), but it's
possible that the problem would start affecting more WADs if we add
support for the vWii common key (which uses index 2). Adding support
for the vWii common key would also prevent us from using the simpler
solution of always forcing the index to 0 if the title is not Korean.
2019-07-21 09:58:16 +02:00
JosJuice 4ee73dbad3 IOS: Put common key handles in an array 2019-07-21 09:37:06 +02:00
JosJuice 34f32898e6 DiscIO: Merge WiiWAD into VolumeWAD
These two classes are very similar, so let's merge them.
2019-07-18 22:29:04 +02:00
JosJuice 0f5a4b37ee DiscIO: Add functions CreateDisc and CreateWAD
...in addition to the existing function CreateVolume
(renamed from CreateVolumeFromFilename).

Lets code easily add constraints such as not letting the user
select a WAD file when using the disc changing functionality.
2019-07-18 22:29:04 +02:00
JosJuice 0b11f15158 Fix VolumeVerifier considering Wii Menu WADs to have wrong region 2019-07-15 12:42:13 +02:00
Lioncash 4f1f55093f Common/MsgHandler: Namespace code within the Common namespace
Closes another gap in the Common library where code isn't being
namespaced under it.
2019-06-19 16:03:55 -04:00
Connor McLaughlin ab66e91766
Merge pull request #8134 from JosJuice/filesystem-string-view
DiscIO: Use std::string_view in FileSystem::FindFileInfo
2019-06-08 20:52:34 +10:00
Lioncash 5512876842 General: Migrate from deprecated mbedTLS functions
As indicated by mbedTLS' documentation, all of the relevant functions
have been superseded by _ret-suffixed variants in mbedTLS version
2.7.0.
2019-06-07 22:51:58 -04:00
Lioncash a9663669dc Common/CommonFuncs: Remove now-unneccessary ArraySize function
Since C++17, non-member std::size() is present in the standard library
which also operates on regular C arrays. Given that, we can just replace
usages of ArraySize with that where applicable.

In many cases, we can just change the actual C array ArraySize() was
called on into a std::array and just use its .size() member function
instead.

In some other cases, we can collapse the loops they were used in, into a
ranged-for loop, eliminating the need for en explicit bounds query.
2019-06-01 10:07:57 -04:00
Lioncash cbcafd6830 DiscIO/CMakeLists: Specify headers in target sources 2019-05-31 06:54:25 -04:00
JosJuice f25367a7a2 DiscIO: Avoid copies when comparing FileSystemGCWii file names 2019-05-29 14:53:39 +02:00
JosJuice d4b069f458 DiscIO: Use std::string_view in FileSystem::FindFileInfo
...and in the functions that call it.
2019-05-29 08:29:56 +02:00
Lioncash d220e33862 DiscIO/VolumeVerifier: Make no-argument overload of GetBiggestUsedOffset() const
The overload taking a partition is already a const member function, so
this one can be turned into one as well.
2019-05-27 10:40:41 -04:00
Lioncash bf6948c1d4 DiscIO/VolumeVerifier: Use structured bindings where applicable
Allows providing better names than "first" or "second".
2019-05-27 10:40:38 -04:00
Lioncash fa57396e97 DiscIO/VolumeVerifier: In-class initialize members where applicable
Removes redundant initializers from the constructor and provides
initializers for all members that don't already have one for consistency
(and deterministic initial state).
2019-05-27 10:40:17 -04:00
Lioncash 52eb2d0d82 DiscIO/VolumeVerifier: Default destructor within the cpp file
Given the volume verifier has quite a few non-trivial object within it,
it's best to default the destructor within the cpp file to prevent
inlining complex destruction logic elsewhere, while also making it nicer
if a forward-declared type is ever used in a member variable.
2019-05-27 10:19:51 -04:00
Lioncash 0ccaa2b5d6 DiscIO/VolumeVerifier: Take std::string by value in AddProblem()
This allows both std::moving into the function and moving the parameter
from within the function, potentially avoiding an unnecessary copy.
2019-05-27 10:17:11 -04:00