Commit Graph

1121 Commits

Author SHA1 Message Date
JosJuice f2c38c0e67 RVZ: Make m_rvz a template parameter
Needed for the next commit.
2020-06-17 13:48:45 +02:00
JosJuice ca4e4a6207 WIA/RVZ: Move (de)compression to a different file 2020-06-17 13:48:45 +02:00
JosJuice 3f753fc87d RVZ: Detect junk data in the same block as a file 2020-06-17 13:48:44 +02:00
JosJuice 4b74993374 RVZ: Store pseudorandom junk data efficiently 2020-06-17 13:48:42 +02:00
JosJuice 1e92b54bf5 WIA/RVZ: Skip some memory allocations when reusing chunks 2020-06-17 13:48:16 +02:00
JosJuice f5ef70fc76 RVZ: Don't store redundant exceptions when chunk size is < 2 MiB 2020-06-17 13:48:16 +02:00
JosJuice b06c50ed2e RVZ: Support chunk sizes between 32 KiB and 2 MiB
WIA doesn't support smaller than 2 MiB.
2020-06-17 13:48:15 +02:00
JosJuice 0d433baeb5 RVZ: Remove PURGE support
PURGE isn't especially useful, while requiring some annoying
special handling in the file format. If you want no compression,
use NONE. If you want fast compression, use Zstandard.
2020-06-17 13:48:15 +02:00
JosJuice 1f7c0b636f RVZ: Add Zstandard as a compression method 2020-06-17 13:48:12 +02:00
JosJuice e2ae2b3b0b Add new file format RVZ based on WIA 2020-06-17 13:47:34 +02:00
JosJuice f21a254042 WIA: Implement multithreaded compression 2020-06-17 13:47:34 +02:00
JosJuice 9dea8169e0 WIA: Write all headers at the start of the file
Gets rid of the need to seek to the end of the file
when opening a file.

The downside of this is that we waste a little space,
since we can't know in advance exactly how much
space the compressed parts of the headers will need.
2020-06-17 13:47:34 +02:00
JosJuice e5b9e1ba1f WIA: Reuse groups when writing
This is useful for the way Dolphin scrubs Wii discs.
The encrypted data is what gets zeroed out, but this
zeroed out data then gets decrypted before being stored,
and the resulting data does not compress well.
However, each block of decrypted scrubbed data is
identical given the same encryption key, and there's
nothing stopping us from making multiple group entries
point to the same offset in the file, so we only have
to store one copy of this data per partition.

For reference, wit zeroes out the decrypted data,
but Dolphin's WIA writer can't do this because it currently
doesn't know which parts of the disc are scrubbed.

This is also useful for things such as storing Datel discs
full of 0x55 blocks (repesenting unreadable blocks)
without compression enabled.
2020-06-17 13:47:34 +02:00
JosJuice 40e46aee57 WIA: Store all-zero data efficiently 2020-06-17 13:47:34 +02:00
JosJuice e8b019ac29 WIA: Implement compression 2020-06-17 13:47:30 +02:00
JosJuice e936c4acd8 WIA: Write hash exceptions 2020-06-17 13:43:54 +02:00
JosJuice 3b8c44fd0e WIA: Decrypt Wii data when writing 2020-06-17 13:43:54 +02:00
JosJuice 115edea34e WIA: Add early support for WIA writing 2020-06-17 13:43:54 +02:00
JosJuice 791e363c9a WIA: Make use of the exception lists 2020-06-17 13:43:54 +02:00
JosJuice 47067f661a WIA: Properly check for overlapping data 2020-06-17 13:43:53 +02:00
JosJuice 04089f24f9 WIA: Implement re-encryption of Wii partition data 2020-06-17 13:43:53 +02:00
JosJuice e3d291a529 WIA: Check the internal WIA hashes 2020-06-17 13:43:53 +02:00
JosJuice 827437c036 WIA: Fix the handling of chunk sizes larger than 2 MiB 2020-06-17 13:43:53 +02:00
JosJuice 0b407228b7 WIA: Add documentation 2020-06-17 13:43:53 +02:00
JosJuice 01a77ae8a1 WIA: Implement caching and partial decompression 2020-06-17 13:43:53 +02:00
JosJuice b59ef81a7e WIA: Implement bzip2, LZMA, and LZMA2 decompression 2020-06-17 13:43:52 +02:00
JosJuice 1579e061a3 WIA: Correctly handle data with size not divisible by chunk size 2020-06-17 13:43:52 +02:00
JosJuice 3c373c8aa8 WIA: Treat groups with size 0 as containing only zeroes 2020-06-17 13:43:52 +02:00
JosJuice 36991e2dde WIA: Implement PURGE decompression 2020-06-17 13:43:52 +02:00
JosJuice 3672bd79f3 WIA: Implement ReadWiiDecrypted 2020-06-17 13:43:52 +02:00
JosJuice 2a5fcc9c25 WIA: Add reading raw data 2020-06-17 13:43:51 +02:00
JosJuice 8da5d0c4fe Add an early version of WIABlobReader
It can currently only read the first 0x80 bytes of a disc image,
which is enough for identifying it but not for doing anything else.
2020-06-17 13:43:51 +02:00
JosJuice 162e3be82b Show an OSD message when running a disc image with a large block size
This is intended to catch WIA files which have been created using
wit's default parameters (40 MiB block size), once the WIA PR is
merged. The check does however also work for GCZ files – not that
I think anyone has a GCZ file with a block size that large.
2020-06-07 14:11:00 +02:00
JosJuice b93bbbf2e5 DiscIO: Implement multithreaded compression 2020-05-06 14:18:51 +02:00
degasus f8f592c345 Externals: Update minizip search path. 2020-04-29 12:41:36 +02:00
JosJuice 5b722b775a VolumeVerifier: Treat IOS9 as always present
Fixes a false positive for Mario Party 8.
2020-04-28 23:40:01 +02:00
JMC47 116cef572b
Merge pull request #8738 from JosJuice/convert-dialog
Replace the compress/uncompress actions with a convert dialog
2020-04-28 12:37:17 -04:00
JosJuice bacf0d629d VolumeVerifier: Show desync warning for dual layer discs too 2020-04-25 19:47:08 +02:00
JosJuice 6ffcbcee70 DiscIO: Move scrubbing code out of ConvertToGCZ
This way, scrubbing can also be performed when converting
to other formats.
2020-04-24 15:11:20 +02:00
JosJuice 04c7892b93 DiscIO: Add GameCube disc scrubbing support
The code was actually already rather well adapted for this.
We more or less just have to skip ParseDisc and run
ParsePartitionData directly. This required the PartitionHeader
struct to be removed (which wasn't that useful anyway).
2020-04-24 15:10:36 +02:00
JosJuice cefc2a7baa DiscIO: Fix edge case where blocks could get scrubbed accidentally
If we start 31 KiB into a 32 KiB block and want to mark 2 KiB
of data as used, we need to mark 2 blocks as used, not just 1.

This problem is avoided when calling MarkAsUsed from
MarkAsUsedE, since MarkAsUsedE aligns to 32 KiB on its own.
Most calls to MarkAsUsed are from MarkAsUsedE, which is why
this hasn't been a noticeable problem in the past.
2020-04-24 15:10:36 +02:00
JosJuice 42f6913bcc Move DiscIO::ConvertToPlain to FileBlob.cpp
There is no longer anything GCZ specific about it.
2020-04-24 15:10:35 +02:00
JosJuice 8a9597e32e DiscIO: Allow converting from formats other than ISO and GCZ
The constant DESIRED_BUFFER_SIZE was determined by multiplying the
old hardcoded value 32 with the default GCZ block size 16 KiB.
Not sure if it actually is the best value, but it seems fine.
2020-04-24 15:10:35 +02:00
JosJuice 432f342bc8 DiscIO: Use a struct for Wii hashes 2020-04-24 14:44:29 +02:00
JosJuice da9e0fb598 DiscIO: Parallelize the re-encryption code 2020-04-24 14:44:26 +02:00
JosJuice 319c508978 DiscIO: Implement re-encryption of Wii partition data 2020-04-24 14:24:12 +02:00
JosJuice a4c7100bcc DiscIO: Use partition data offset for ReadWiiDecrypted parameter
Instead of the partition offset (which is usually 0x20000 less).
2020-04-24 14:16:55 +02:00
JosJuice 19e9a9c945 DiscIO: Clean up decompression size calculation
We can use subtraction and std::min instead of
modulo and explicit if statements.

This commit does not change the behavior.
2020-04-15 22:15:40 +02:00
JosJuice 3aa463cdae DiscIO: Fix decompressing writing too much sometimes
This issue cannot happen with good dumps due to their size,
but it can happen with trimmed dumps.
2020-04-12 21:47:10 +02:00
JosJuice 26b21e3186 DiscIO: Fix decompressing writing too little sometimes
This issue cannot happen with good dumps due to their size,
but it can happen with trimmed dumps.
2020-04-12 21:45:55 +02:00