Commit Graph

33102 Commits

Author SHA1 Message Date
JosJuice 3f753fc87d RVZ: Detect junk data in the same block as a file 2020-06-17 13:48:44 +02:00
JosJuice 4b74993374 RVZ: Store pseudorandom junk data efficiently 2020-06-17 13:48:42 +02:00
JosJuice 1e92b54bf5 WIA/RVZ: Skip some memory allocations when reusing chunks 2020-06-17 13:48:16 +02:00
JosJuice f5ef70fc76 RVZ: Don't store redundant exceptions when chunk size is < 2 MiB 2020-06-17 13:48:16 +02:00
JosJuice b06c50ed2e RVZ: Support chunk sizes between 32 KiB and 2 MiB
WIA doesn't support smaller than 2 MiB.
2020-06-17 13:48:15 +02:00
JosJuice 0d433baeb5 RVZ: Remove PURGE support
PURGE isn't especially useful, while requiring some annoying
special handling in the file format. If you want no compression,
use NONE. If you want fast compression, use Zstandard.
2020-06-17 13:48:15 +02:00
JosJuice 1f7c0b636f RVZ: Add Zstandard as a compression method 2020-06-17 13:48:12 +02:00
JosJuice e2ae2b3b0b Add new file format RVZ based on WIA 2020-06-17 13:47:34 +02:00
JosJuice f21a254042 WIA: Implement multithreaded compression 2020-06-17 13:47:34 +02:00
JosJuice 9dea8169e0 WIA: Write all headers at the start of the file
Gets rid of the need to seek to the end of the file
when opening a file.

The downside of this is that we waste a little space,
since we can't know in advance exactly how much
space the compressed parts of the headers will need.
2020-06-17 13:47:34 +02:00
JosJuice e5b9e1ba1f WIA: Reuse groups when writing
This is useful for the way Dolphin scrubs Wii discs.
The encrypted data is what gets zeroed out, but this
zeroed out data then gets decrypted before being stored,
and the resulting data does not compress well.
However, each block of decrypted scrubbed data is
identical given the same encryption key, and there's
nothing stopping us from making multiple group entries
point to the same offset in the file, so we only have
to store one copy of this data per partition.

For reference, wit zeroes out the decrypted data,
but Dolphin's WIA writer can't do this because it currently
doesn't know which parts of the disc are scrubbed.

This is also useful for things such as storing Datel discs
full of 0x55 blocks (repesenting unreadable blocks)
without compression enabled.
2020-06-17 13:47:34 +02:00
JosJuice 40e46aee57 WIA: Store all-zero data efficiently 2020-06-17 13:47:34 +02:00
JosJuice e8b019ac29 WIA: Implement compression 2020-06-17 13:47:30 +02:00
JosJuice e936c4acd8 WIA: Write hash exceptions 2020-06-17 13:43:54 +02:00
JosJuice 3b8c44fd0e WIA: Decrypt Wii data when writing 2020-06-17 13:43:54 +02:00
JosJuice 115edea34e WIA: Add early support for WIA writing 2020-06-17 13:43:54 +02:00
JosJuice 791e363c9a WIA: Make use of the exception lists 2020-06-17 13:43:54 +02:00
JosJuice 47067f661a WIA: Properly check for overlapping data 2020-06-17 13:43:53 +02:00
JosJuice 04089f24f9 WIA: Implement re-encryption of Wii partition data 2020-06-17 13:43:53 +02:00
JosJuice e3d291a529 WIA: Check the internal WIA hashes 2020-06-17 13:43:53 +02:00
JosJuice 827437c036 WIA: Fix the handling of chunk sizes larger than 2 MiB 2020-06-17 13:43:53 +02:00
JosJuice 0b407228b7 WIA: Add documentation 2020-06-17 13:43:53 +02:00
JosJuice 01a77ae8a1 WIA: Implement caching and partial decompression 2020-06-17 13:43:53 +02:00
JosJuice b59ef81a7e WIA: Implement bzip2, LZMA, and LZMA2 decompression 2020-06-17 13:43:52 +02:00
JosJuice 1579e061a3 WIA: Correctly handle data with size not divisible by chunk size 2020-06-17 13:43:52 +02:00
JosJuice 3c373c8aa8 WIA: Treat groups with size 0 as containing only zeroes 2020-06-17 13:43:52 +02:00
JosJuice 36991e2dde WIA: Implement PURGE decompression 2020-06-17 13:43:52 +02:00
JosJuice 3672bd79f3 WIA: Implement ReadWiiDecrypted 2020-06-17 13:43:52 +02:00
JosJuice 2a5fcc9c25 WIA: Add reading raw data 2020-06-17 13:43:51 +02:00
JosJuice 8da5d0c4fe Add an early version of WIABlobReader
It can currently only read the first 0x80 bytes of a disc image,
which is enough for identifying it but not for doing anything else.
2020-06-17 13:43:51 +02:00
Tilka 0bc7dcd172
Merge pull request #8882 from JosJuice/convert-dropdown-one-option
DolphinQt: Disable convert dialog dropdowns when they have one option
2020-06-17 12:38:19 +01:00
JosJuice b45f2c2ad3 DolphinQt: Disable convert dialog dropdowns when they have one option 2020-06-17 13:23:58 +02:00
Tilka 3411802fd1
Merge pull request #8881 from JosJuice/convert-same-format
DolphinQt: Allow converting to same format
2020-06-17 11:56:49 +01:00
JosJuice ec3ea6c1df DolphinQt: Allow converting to same format
Mainly useful for WIA and RVZ, but also has some use for GCZ.
2020-06-17 12:44:16 +02:00
JosJuice 864dcd40b5 Revert "DolphinQt: Disable converting from TGC"
This reverts commit d006a8b52f.
2020-06-17 12:32:40 +02:00
JosJuice 31ecc6c214 Revert "DolphinQt: Actually disable converting from TGC"
This reverts commit c236d89f64.
2020-06-17 12:32:39 +02:00
JosJuice 75d032161f DiscIO: Rework the implementation of TGC reading
Fixes https://bugs.dolphin-emu.org/issues/10654.

To quote the documenation file included with the program tgctogcm:

"TGC's are miniaturized .gcm images with a 32kB header.
The embedded gcm contains some bogus data, namely:
-FST Location (0x424 in gcm)
-DOL Location (0x420 in gcm)
-FST File offsets (all files are offset/spoofed by a certain amount)"

Dolphin has been handling the values at 0x420 and 0x424 by simply
overwriting them with a working value (just like tgctogcm does),
but it has used a different approach for the file offsets in the FST.
Instead of changing the offsets that are stored in the FST, Dolphin
changed where the files actually are placed on the virtual disc.
My hope was that this would make the loading times more accurate to
how they are when running a TGC file as part of a larger disc.
However, there are TGC files where we would need to move files
backwards on the disc in order to do this (this is what issue
10654 is about), so the approach we have been using is flawed.

This change makes Dolphin overwrite offsets in the FST instead, like
tgctogcm does. Other than making Dolphin handle the affected TGC files
correctly, this change also makes it so that unnecessary padding data
isn't written if you use Dolphin to convert a TGC file to an ISO file.
This feature is not actually implemented in Dolphin as of now, but I'm
planning to add it in the near future as part of a larger feature.
2020-06-17 12:32:39 +02:00
Admiral H. Curtiss 476c95900d EXI: When loading a savestate with a mismatching GCI folder memory card, reinizialize it with the header from the savestate to let a game still recognize it as the same card. 2020-06-17 01:44:46 +02:00
Admiral H. Curtiss 8b13e1882a EXI: Store data for regenerating a GCI folder memory card header in EXI_Channel and pass it down to the memory card device. 2020-06-17 01:44:46 +02:00
Admiral H. Curtiss e810d492f2 GCMemcard: Split off HeaderData from Header to have a compact block of data for consistent initialization. 2020-06-17 01:44:46 +02:00
Admiral H. Curtiss cc52558c0e GCMemcard: Assert struct requirements in header. 2020-06-17 01:30:42 +02:00
Admiral H. Curtiss 07bd200596 GCMemcard: Remove dependency on g_SRAM and force the caller to provide the relevant values instead. 2020-06-17 01:30:37 +02:00
Markus Wick 7cdd0e2430
Merge pull request #8817 from Tilka/cmake
CMake: support using system-wide zstd library
2020-06-16 22:55:46 +02:00
JosJuice fba5ff5ea5 Translation resources sync with Transifex 2020-06-16 22:51:59 +02:00
Tilka 1f505870a9
Merge pull request #8873 from AdmiralCurtiss/gcmemcard-namespace
GCMemcard: Move into a Memcard namespace.
2020-06-16 19:55:06 +01:00
Tillmann Karras 506ae2f6b1 Externals/zstd: fix CMake project name 2020-06-16 19:42:55 +01:00
Tillmann Karras 24eee74937 CMake: support using system-wide zstd library
CMake does not have native support for zstd yet.
2020-06-16 19:42:55 +01:00
Tilka de5941d6f8
Merge pull request #8878 from JosJuice/actually-disable-tgc-conversion
DolphinQt: Actually disable converting from TGC
2020-06-16 19:39:40 +01:00
Tilka a2b7627d66
Merge pull request #8860 from Ebola16/Emu
Android: Disallow multiple launcher activities
2020-06-16 19:38:55 +01:00
Admiral H. Curtiss 914ebdf0dd GCMemcard: Move from global namespace into a Memcard namespace. 2020-06-16 20:18:39 +02:00