A file format that puts a memory-mappable metadata header in front of arbitrary contents. More...
Classes | |
| class | sourcemeta::core::MetapackError |
| struct | sourcemeta::core::MetapackHeader |
| struct | sourcemeta::core::MetapackView |
Enumerations | |
| enum class | sourcemeta::core::MetapackEncoding : std::uint8_t { MetapackEncoding::Identity = 0 , MetapackEncoding::GZIP = 1 } |
Functions | |
| SOURCEMETA_CORE_METAPACK_EXPORT auto | sourcemeta::core::metapack_parse (const FileView &view) -> std::optional< MetapackView > |
| SOURCEMETA_CORE_METAPACK_EXPORT auto | sourcemeta::core::metapack_read_json (const std::filesystem::path &path, const std::uint64_t maximum_decompression_ratio=METAPACK_MAXIMUM_DECOMPRESSION_RATIO) -> std::optional< JSON > |
| SOURCEMETA_CORE_METAPACK_EXPORT auto | sourcemeta::core::metapack_read_json (const FileView &view, const std::uint64_t maximum_decompression_ratio=METAPACK_MAXIMUM_DECOMPRESSION_RATIO) -> std::optional< JSON > |
| SOURCEMETA_CORE_METAPACK_EXPORT auto | sourcemeta::core::metapack_read_json (const MetapackView &parsed, const std::uint64_t maximum_decompression_ratio=METAPACK_MAXIMUM_DECOMPRESSION_RATIO) -> std::optional< JSON > |
| SOURCEMETA_CORE_METAPACK_EXPORT auto | sourcemeta::core::metapack_checksum_hex (const MetapackHeader &header) -> std::array< char, METAPACK_CHECKSUM_HEX_LENGTH > |
| SOURCEMETA_CORE_METAPACK_EXPORT auto | sourcemeta::core::metapack_last_modified (const MetapackHeader &header) -> std::chrono::system_clock::time_point |
| template<typename T> | |
| auto | sourcemeta::core::metapack_extension_as (const std::span< const std::uint8_t > extension) -> const T * |
| SOURCEMETA_CORE_METAPACK_EXPORT auto | sourcemeta::core::metapack_write_json (const std::filesystem::path &destination, const JSON &document, const std::string_view media_type, const MetapackEncoding encoding, const std::span< const std::uint8_t > extension={}) -> void |
| SOURCEMETA_CORE_METAPACK_EXPORT auto | sourcemeta::core::metapack_write_json_pretty (const std::filesystem::path &destination, const JSON &document, const std::string_view media_type, const MetapackEncoding encoding, const std::span< const std::uint8_t > extension={}) -> void |
| SOURCEMETA_CORE_METAPACK_EXPORT auto | sourcemeta::core::metapack_write_text (const std::filesystem::path &destination, const std::string_view contents, const std::string_view media_type, const MetapackEncoding encoding, const std::span< const std::uint8_t > extension={}) -> void |
| SOURCEMETA_CORE_METAPACK_EXPORT auto | sourcemeta::core::metapack_write_bytes (const std::filesystem::path &destination, const std::span< const std::uint8_t > contents, const std::string_view media_type, const MetapackEncoding encoding, const std::span< const std::uint8_t > extension={}) -> void |
| SOURCEMETA_CORE_METAPACK_EXPORT auto | sourcemeta::core::metapack_write_file (const std::filesystem::path &destination, const std::filesystem::path &source, const std::string_view media_type, const MetapackEncoding encoding, const std::span< const std::uint8_t > extension={}) -> void |
Variables | |
| constexpr std::size_t | sourcemeta::core::METAPACK_CHECKSUM_HEX_LENGTH {64} |
| constexpr std::uint16_t | sourcemeta::core::METAPACK_MAXIMUM_MEDIA_TYPE_BYTES |
| constexpr std::uint32_t | sourcemeta::core::METAPACK_MAXIMUM_EXTENSION_BYTES |
| constexpr std::uint64_t | sourcemeta::core::METAPACK_MAXIMUM_DECOMPRESSION_RATIO {1024} |
A file format that puts a memory-mappable metadata header in front of arbitrary contents.
A filesystem answers a few questions about a file without anyone opening it, which is what a stat call is for: how large it is, when it last changed, what kind of file it is. Everything else a file has to say about itself has nowhere to live. Keeping it beside the file, in a sidecar, a database, or a naming convention, costs a second lookup and lets the two drift apart. Keeping it inside the contents means decoding the contents to reach it.
A Metapack file carries it in front of the contents instead, as a packed little-endian structure whose fields sit at offsets fixed at compile time. A reader memory maps the file, points the header structure at the start of the mapping, and has every field without a read, a copy, or an allocation. What follows is handed back as a view into that same mapping, so the contents cost nothing until something reaches for them.
What the header holds, field by field, is the sourcemeta::core::MetapackHeader structure itself. After it come the media type, then an extension area, then the contents, each located from the length recorded before it.
The extension area carries whatever else a given kind of file needs to say about itself, as a packed structure that the writer defines and that a reader interprets in place, exactly like the header itself.
This functionality is included as follows:
| class sourcemeta::core::MetapackError |
An error that represents contents that cannot be written as a file of this format
Public Member Functions | |
|---|---|
| MetapackError (const StaticString message) noexcept | |
| Construct an error with the given message. | |
| Public Member Functions inherited from sourcemeta::core::Error | |
| auto | message () const noexcept -> StaticString |
Additional Inherited Members | |
| Protected Member Functions inherited from sourcemeta::core::Error | |
| Error (const StaticString message) noexcept | |
| Construct the error given its message. | |
| struct sourcemeta::core::MetapackHeader |
The metadata that a file of this format begins with
Public Attributes | |
|---|---|
| std::uint32_t | magic {0} |
| The magic number that identifies the format. | |
| std::uint16_t | format_version {0} |
| The revision of the format the file was written to. | |
| MetapackEncoding | encoding {MetapackEncoding::Identity} |
| The encoding the contents are stored under. | |
| std::uint8_t | reserved {0} |
| Reserved for a future revision of the format. | |
| std::int64_t | last_modified {0} |
| Nanoseconds since the Unix epoch at which the file was written. | |
| std::uint64_t | content_bytes {0} |
| How many bytes the contents take once decoded. | |
| std::array< std::uint8_t, 32 > | checksum {} |
| std::uint16_t | media_type_length {0} |
| How many bytes the media type that follows the header takes. | |
| std::array<std::uint8_t, 32> sourcemeta::core::MetapackHeader::checksum {} |
The SHA-256 digest of the decoded contents, which names this revision of them for a cache to compare against rather than being checked on read
| struct sourcemeta::core::MetapackView |
Every part of a file, located in a single pass. The views point into the memory-mapped file, so they last exactly as long as it stays mapped
Public Attributes | |
|---|---|
| const MetapackHeader * | header {nullptr} |
| The metadata the file begins with. | |
| std::string_view | media_type |
| The media type to serve the contents as. | |
| std::span< const std::uint8_t > | extension |
| The extension area, as the bytes the writer put there. | |
| std::span< const std::uint8_t > | payload |
| The contents, as stored. | |
|
strong |
| SOURCEMETA_CORE_METAPACK_EXPORT auto sourcemeta::core::metapack_checksum_hex | ( | const MetapackHeader & | header | ) | -> std::array< char, METAPACK_CHECKSUM_HEX_LENGTH > |
Render the checksum a header records as lowercase hexadecimal. For example:
| auto sourcemeta::core::metapack_extension_as | ( | const std::span< const std::uint8_t > | extension | ) | -> const T * |
What an extension area holds, as the layout the writer of the file put there, or nothing when it is too small to hold one. For example:
| SOURCEMETA_CORE_METAPACK_EXPORT auto sourcemeta::core::metapack_last_modified | ( | const MetapackHeader & | header | ) | -> std::chrono::system_clock::time_point |
The point in time a header records. For example:
| SOURCEMETA_CORE_METAPACK_EXPORT auto sourcemeta::core::metapack_parse | ( | const FileView & | view | ) | -> std::optional< MetapackView > |
Locate every part of a memory-mapped file. For example:
| SOURCEMETA_CORE_METAPACK_EXPORT auto sourcemeta::core::metapack_read_json | ( | const FileView & | view, |
| const std::uint64_t | maximum_decompression_ratio = METAPACK_MAXIMUM_DECOMPRESSION_RATIO ) -> std::optional< JSON > |
Read the contents of a memory-mapped file as a JSON document. For example:
| SOURCEMETA_CORE_METAPACK_EXPORT auto sourcemeta::core::metapack_read_json | ( | const MetapackView & | parsed, |
| const std::uint64_t | maximum_decompression_ratio = METAPACK_MAXIMUM_DECOMPRESSION_RATIO ) -> std::optional< JSON > |
Read the contents of an already located file as a JSON document. For example:
| SOURCEMETA_CORE_METAPACK_EXPORT auto sourcemeta::core::metapack_read_json | ( | const std::filesystem::path & | path, |
| const std::uint64_t | maximum_decompression_ratio = METAPACK_MAXIMUM_DECOMPRESSION_RATIO ) -> std::optional< JSON > |
Read the contents of a file at the given path as a JSON document. For example:
| SOURCEMETA_CORE_METAPACK_EXPORT auto sourcemeta::core::metapack_write_bytes | ( | const std::filesystem::path & | destination, |
| const std::span< const std::uint8_t > | contents, | ||
| const std::string_view | media_type, | ||
| const MetapackEncoding | encoding, | ||
| const std::span< const std::uint8_t > | extension = {} ) -> void |
Write bytes as the contents of a file. A media type or extension too large for the format to record throws. For example:
| SOURCEMETA_CORE_METAPACK_EXPORT auto sourcemeta::core::metapack_write_file | ( | const std::filesystem::path & | destination, |
| const std::filesystem::path & | source, | ||
| const std::string_view | media_type, | ||
| const MetapackEncoding | encoding, | ||
| const std::span< const std::uint8_t > | extension = {} ) -> void |
Write another file as the contents of a file. A media type or extension too large for the format to record throws. For example:
| SOURCEMETA_CORE_METAPACK_EXPORT auto sourcemeta::core::metapack_write_json | ( | const std::filesystem::path & | destination, |
| const JSON & | document, | ||
| const std::string_view | media_type, | ||
| const MetapackEncoding | encoding, | ||
| const std::span< const std::uint8_t > | extension = {} ) -> void |
Write a JSON document as the contents of a file. A media type or extension too large for the format to record throws. For example:
| SOURCEMETA_CORE_METAPACK_EXPORT auto sourcemeta::core::metapack_write_json_pretty | ( | const std::filesystem::path & | destination, |
| const JSON & | document, | ||
| const std::string_view | media_type, | ||
| const MetapackEncoding | encoding, | ||
| const std::span< const std::uint8_t > | extension = {} ) -> void |
Write a JSON document as the contents of a file, indented for a human to read. A media type or extension too large for the format to record throws. For example:
| SOURCEMETA_CORE_METAPACK_EXPORT auto sourcemeta::core::metapack_write_text | ( | const std::filesystem::path & | destination, |
| const std::string_view | contents, | ||
| const std::string_view | media_type, | ||
| const MetapackEncoding | encoding, | ||
| const std::span< const std::uint8_t > | extension = {} ) -> void |
Write text as the contents of a file, terminated by a line feed. A media type or extension too large for the format to record throws. For example:
|
inlineconstexpr |
How many characters it takes to render a checksum
|
inlineconstexpr |
How far stored contents may inflate before a reader refuses them, so that a small file cannot ask for an unbounded amount of memory
|
inlineconstexpr |
The largest extension area a file of this format can carry
|
inlineconstexpr |
The longest media type a file of this format can declare