Commit Graph

51 Commits

Author SHA1 Message Date
Léo Lam b7f931fc6b
Merge pull request #9570 from JosJuice/wia-partial-exception-list
DiscIO: Fix reading certain WIA chunks with many exceptions
2021-03-16 11:10:12 +01:00
JosJuice 49ccc77ebb DiscIO: Move some code from DiscExtractor to new file DiscUtils 2021-03-09 20:34:24 +01:00
JosJuice 14bfc0be78 DiscIO: Fix reading certain WIA chunks with many exceptions
The loop in WIARVZFileReader::Chunk::Read could terminate
prematurely if the size argument was smaller than the size
of an exception list which had only been partially loaded.
2021-03-07 14:14:45 +01:00
Shawn Hoffman 84128d9532 rename Common/File to Common/IOFile 2021-01-27 14:29:48 -08:00
JosJuice 9a01c3fb9f Use positional arguments in all translatable fmt strings
We want to use positional arguments in translatable strings
that have more than one argument so that translators can change
the order of them, but the question is: Should we also use
positional arguments in translatable strings with only one
argument? I think it makes most sense that way, partially
so that translators don't even have to be aware of the
non-positional syntax and partially because "translatable
strings use positional arguments" is an easier rule for us
to remember than "transitional strings which have more than
one argument use positional arguments". But let me know if
you have a different opinion.
2020-11-16 13:28:11 +01:00
Lioncash 689eec5304 DiscIO: Make use of fmt-capable panic alerts
Migrates the DiscIO code over to fmt.
2020-11-11 01:09:42 -05:00
Lioncash e93fbb7c5e DiscIO: Migrate logging over to fmt
Eliminates quite a bit of the PRI* macros used for handling 64-bit
values.
2020-10-22 15:41:42 -04:00
JosJuice 1084cfec1a Add helper function for fmt::format(GetStringT(... 2020-10-19 14:36:08 +02:00
JosJuice d100c1dc37 DiscIO: Make use of fmt where applicable
Once nice benefit of fmt is that we can use positional arguments
in localizable strings. This a feature which has been
requested for the Korean translation of strings like
"Errors were found in %zu blocks in the %s partition."
and which will no doubt be useful for other languages too.
2020-10-19 14:30:47 +02:00
JosJuice 3feea108db DiscIO: Decrease RAM usage during zstd compression
By calling ZSTD_CCtx_setPledgedSrcSize, we can let zstd know
how large a chunk is going to be before which start compressing
it, which lets zstd avoid allocating more memory than needed
for various internal buffers. This greatly reduces the RAM usage
when using a high compression level with a small chunk size,
and doesn't have much of an effect in other circumstances.

A side effect of calling ZSTD_CCtx_setPledgedSrcSize is that
zstd by default will write the uncompressed size into the
compressed data stream as metadata. In order to save space,
and since the decompressed size can be figured out through
the structure of the RVZ format anyway, we disable writing
the uncompressed size by setting ZSTD_c_contentSizeFlag to 0.
2020-10-07 16:40:06 +02:00
JosJuice ca46028cde DiscIO: Use std::function for compression callback 2020-09-16 18:36:50 +02:00
LC db067104ed
Merge pull request #9047 from JosJuice/wia-header-heuristic
DiscIO: Adjust WIA/RVZ header size heuristic
2020-09-01 17:58:51 -04:00
JosJuice bdd28c0edf DiscIO: Expand WIA/RVZ header size heuristic comments 2020-08-31 14:46:13 +02:00
JosJuice 047bb80337 DiscIO: Add parameters to BlobReader::SupportsReadWiiDecrypted
It's possible (but rare) for a WIA or RVZ file to support
this for some partitions but not all, and for the game and
the blob code to disagree on how large a partition is.
2020-08-29 15:18:52 +02:00
JosJuice f7f0fa2d85 DiscIO: Adjust WIA/RVZ header size heuristic
The heuristic was not allocating enough space for Metroid: Other M,
at least when using the default settings. (This didn't break the
file, it just caused some headers to be placed at the end of the
file instead of at the start and wasted a few hundred kilobytes.)
2020-08-22 22:26:55 +02:00
Tillmann Karras d2b06f4786 Fix various -Wshadow warnings 2020-08-08 21:45:03 +01:00
JosJuice a05b1f9ab9 DiscIO: Fix conversion warning
Should fix https://bugs.dolphin-emu.org/issues/12208.
2020-07-29 10:55:02 +02:00
Tillmann Karras bf5b1092b2 DiscIO: fix -Wreorder warning 2020-07-24 17:41:37 +01:00
JosJuice d494e0230c Show file format details in game properties 2020-06-21 20:47:23 +02:00
JosJuice 224c6e799d RVZ: Extend GroupEntry 2020-06-17 13:48:45 +02:00
JosJuice f2c38c0e67 RVZ: Make m_rvz a template parameter
Needed for the next commit.
2020-06-17 13:48:45 +02:00
JosJuice ca4e4a6207 WIA/RVZ: Move (de)compression to a different file 2020-06-17 13:48:45 +02:00
JosJuice 3f753fc87d RVZ: Detect junk data in the same block as a file 2020-06-17 13:48:44 +02:00
JosJuice 4b74993374 RVZ: Store pseudorandom junk data efficiently 2020-06-17 13:48:42 +02:00
JosJuice 1e92b54bf5 WIA/RVZ: Skip some memory allocations when reusing chunks 2020-06-17 13:48:16 +02:00
JosJuice f5ef70fc76 RVZ: Don't store redundant exceptions when chunk size is < 2 MiB 2020-06-17 13:48:16 +02:00
JosJuice b06c50ed2e RVZ: Support chunk sizes between 32 KiB and 2 MiB
WIA doesn't support smaller than 2 MiB.
2020-06-17 13:48:15 +02:00
JosJuice 0d433baeb5 RVZ: Remove PURGE support
PURGE isn't especially useful, while requiring some annoying
special handling in the file format. If you want no compression,
use NONE. If you want fast compression, use Zstandard.
2020-06-17 13:48:15 +02:00
JosJuice 1f7c0b636f RVZ: Add Zstandard as a compression method 2020-06-17 13:48:12 +02:00
JosJuice e2ae2b3b0b Add new file format RVZ based on WIA 2020-06-17 13:47:34 +02:00
JosJuice f21a254042 WIA: Implement multithreaded compression 2020-06-17 13:47:34 +02:00
JosJuice 9dea8169e0 WIA: Write all headers at the start of the file
Gets rid of the need to seek to the end of the file
when opening a file.

The downside of this is that we waste a little space,
since we can't know in advance exactly how much
space the compressed parts of the headers will need.
2020-06-17 13:47:34 +02:00
JosJuice e5b9e1ba1f WIA: Reuse groups when writing
This is useful for the way Dolphin scrubs Wii discs.
The encrypted data is what gets zeroed out, but this
zeroed out data then gets decrypted before being stored,
and the resulting data does not compress well.
However, each block of decrypted scrubbed data is
identical given the same encryption key, and there's
nothing stopping us from making multiple group entries
point to the same offset in the file, so we only have
to store one copy of this data per partition.

For reference, wit zeroes out the decrypted data,
but Dolphin's WIA writer can't do this because it currently
doesn't know which parts of the disc are scrubbed.

This is also useful for things such as storing Datel discs
full of 0x55 blocks (repesenting unreadable blocks)
without compression enabled.
2020-06-17 13:47:34 +02:00
JosJuice 40e46aee57 WIA: Store all-zero data efficiently 2020-06-17 13:47:34 +02:00
JosJuice e8b019ac29 WIA: Implement compression 2020-06-17 13:47:30 +02:00
JosJuice e936c4acd8 WIA: Write hash exceptions 2020-06-17 13:43:54 +02:00
JosJuice 3b8c44fd0e WIA: Decrypt Wii data when writing 2020-06-17 13:43:54 +02:00
JosJuice 115edea34e WIA: Add early support for WIA writing 2020-06-17 13:43:54 +02:00
JosJuice 791e363c9a WIA: Make use of the exception lists 2020-06-17 13:43:54 +02:00
JosJuice 47067f661a WIA: Properly check for overlapping data 2020-06-17 13:43:53 +02:00
JosJuice 04089f24f9 WIA: Implement re-encryption of Wii partition data 2020-06-17 13:43:53 +02:00
JosJuice e3d291a529 WIA: Check the internal WIA hashes 2020-06-17 13:43:53 +02:00
JosJuice 827437c036 WIA: Fix the handling of chunk sizes larger than 2 MiB 2020-06-17 13:43:53 +02:00
JosJuice 01a77ae8a1 WIA: Implement caching and partial decompression 2020-06-17 13:43:53 +02:00
JosJuice b59ef81a7e WIA: Implement bzip2, LZMA, and LZMA2 decompression 2020-06-17 13:43:52 +02:00
JosJuice 1579e061a3 WIA: Correctly handle data with size not divisible by chunk size 2020-06-17 13:43:52 +02:00
JosJuice 3c373c8aa8 WIA: Treat groups with size 0 as containing only zeroes 2020-06-17 13:43:52 +02:00
JosJuice 36991e2dde WIA: Implement PURGE decompression 2020-06-17 13:43:52 +02:00
JosJuice 3672bd79f3 WIA: Implement ReadWiiDecrypted 2020-06-17 13:43:52 +02:00
JosJuice 2a5fcc9c25 WIA: Add reading raw data 2020-06-17 13:43:51 +02:00