Sourcemeta Core 0.0.0
Loading...
Searching...
No Matches
Metapack

A file format that puts a memory-mappable metadata header in front of arbitrary contents. More...

Classes

class  sourcemeta::core::MetapackError
struct  sourcemeta::core::MetapackHeader
struct  sourcemeta::core::MetapackView

Enumerations

enum class  sourcemeta::core::MetapackEncoding : std::uint8_t { MetapackEncoding::Identity = 0 , MetapackEncoding::GZIP = 1 }

Functions

SOURCEMETA_CORE_METAPACK_EXPORT auto sourcemeta::core::metapack_parse (const FileView &view) -> std::optional< MetapackView >
SOURCEMETA_CORE_METAPACK_EXPORT auto sourcemeta::core::metapack_read_json (const std::filesystem::path &path, const std::uint64_t maximum_decompression_ratio=METAPACK_MAXIMUM_DECOMPRESSION_RATIO) -> std::optional< JSON >
SOURCEMETA_CORE_METAPACK_EXPORT auto sourcemeta::core::metapack_read_json (const FileView &view, const std::uint64_t maximum_decompression_ratio=METAPACK_MAXIMUM_DECOMPRESSION_RATIO) -> std::optional< JSON >
SOURCEMETA_CORE_METAPACK_EXPORT auto sourcemeta::core::metapack_read_json (const MetapackView &parsed, const std::uint64_t maximum_decompression_ratio=METAPACK_MAXIMUM_DECOMPRESSION_RATIO) -> std::optional< JSON >
SOURCEMETA_CORE_METAPACK_EXPORT auto sourcemeta::core::metapack_checksum_hex (const MetapackHeader &header) -> std::array< char, METAPACK_CHECKSUM_HEX_LENGTH >
SOURCEMETA_CORE_METAPACK_EXPORT auto sourcemeta::core::metapack_last_modified (const MetapackHeader &header) -> std::chrono::system_clock::time_point
template<typename T>
auto sourcemeta::core::metapack_extension_as (const std::span< const std::uint8_t > extension) -> const T *
SOURCEMETA_CORE_METAPACK_EXPORT auto sourcemeta::core::metapack_write_json (const std::filesystem::path &destination, const JSON &document, const std::string_view media_type, const MetapackEncoding encoding, const std::span< const std::uint8_t > extension={}) -> void
SOURCEMETA_CORE_METAPACK_EXPORT auto sourcemeta::core::metapack_write_json_pretty (const std::filesystem::path &destination, const JSON &document, const std::string_view media_type, const MetapackEncoding encoding, const std::span< const std::uint8_t > extension={}) -> void
SOURCEMETA_CORE_METAPACK_EXPORT auto sourcemeta::core::metapack_write_text (const std::filesystem::path &destination, const std::string_view contents, const std::string_view media_type, const MetapackEncoding encoding, const std::span< const std::uint8_t > extension={}) -> void
SOURCEMETA_CORE_METAPACK_EXPORT auto sourcemeta::core::metapack_write_bytes (const std::filesystem::path &destination, const std::span< const std::uint8_t > contents, const std::string_view media_type, const MetapackEncoding encoding, const std::span< const std::uint8_t > extension={}) -> void
SOURCEMETA_CORE_METAPACK_EXPORT auto sourcemeta::core::metapack_write_file (const std::filesystem::path &destination, const std::filesystem::path &source, const std::string_view media_type, const MetapackEncoding encoding, const std::span< const std::uint8_t > extension={}) -> void

Variables

constexpr std::size_t sourcemeta::core::METAPACK_CHECKSUM_HEX_LENGTH {64}
constexpr std::uint16_t sourcemeta::core::METAPACK_MAXIMUM_MEDIA_TYPE_BYTES
constexpr std::uint32_t sourcemeta::core::METAPACK_MAXIMUM_EXTENSION_BYTES
constexpr std::uint64_t sourcemeta::core::METAPACK_MAXIMUM_DECOMPRESSION_RATIO {1024}

Detailed Description

A file format that puts a memory-mappable metadata header in front of arbitrary contents.

A filesystem answers a few questions about a file without anyone opening it, which is what a stat call is for: how large it is, when it last changed, what kind of file it is. Everything else a file has to say about itself has nowhere to live. Keeping it beside the file, in a sidecar, a database, or a naming convention, costs a second lookup and lets the two drift apart. Keeping it inside the contents means decoding the contents to reach it.

A Metapack file carries it in front of the contents instead, as a packed little-endian structure whose fields sit at offsets fixed at compile time. A reader memory maps the file, points the header structure at the start of the mapping, and has every field without a read, a copy, or an allocation. What follows is handed back as a view into that same mapping, so the contents cost nothing until something reaches for them.

What the header holds, field by field, is the sourcemeta::core::MetapackHeader structure itself. After it come the media type, then an extension area, then the contents, each located from the length recorded before it.

The extension area carries whatever else a given kind of file needs to say about itself, as a packed structure that the writer defines and that a reader interprets in place, exactly like the header itself.

This functionality is included as follows:

#include <sourcemeta/core/metapack.h>

Class Documentation

◆ sourcemeta::core::MetapackError

class sourcemeta::core::MetapackError

An error that represents contents that cannot be written as a file of this format

Inheritance diagram for sourcemeta::core::MetapackError:
sourcemeta::core::Error

Public Member Functions

 MetapackError (const StaticString message) noexcept
 Construct an error with the given message.
Public Member Functions inherited from sourcemeta::core::Error
auto message () const noexcept -> StaticString

Additional Inherited Members

Protected Member Functions inherited from sourcemeta::core::Error
 Error (const StaticString message) noexcept
 Construct the error given its message.

◆ sourcemeta::core::MetapackHeader

struct sourcemeta::core::MetapackHeader

The metadata that a file of this format begins with

Public Attributes

std::uint32_t magic {0}
 The magic number that identifies the format.
std::uint16_t format_version {0}
 The revision of the format the file was written to.
MetapackEncoding encoding {MetapackEncoding::Identity}
 The encoding the contents are stored under.
std::uint8_t reserved {0}
 Reserved for a future revision of the format.
std::int64_t last_modified {0}
 Nanoseconds since the Unix epoch at which the file was written.
std::uint64_t content_bytes {0}
 How many bytes the contents take once decoded.
std::array< std::uint8_t, 32 > checksum {}
std::uint16_t media_type_length {0}
 How many bytes the media type that follows the header takes.

Member Data Documentation

◆ checksum

std::array<std::uint8_t, 32> sourcemeta::core::MetapackHeader::checksum {}

The SHA-256 digest of the decoded contents, which names this revision of them for a cache to compare against rather than being checked on read

◆ sourcemeta::core::MetapackView

struct sourcemeta::core::MetapackView

Every part of a file, located in a single pass. The views point into the memory-mapped file, so they last exactly as long as it stays mapped

Public Attributes

const MetapackHeader * header {nullptr}
 The metadata the file begins with.
std::string_view media_type
 The media type to serve the contents as.
std::span< const std::uint8_t > extension
 The extension area, as the bytes the writer put there.
std::span< const std::uint8_t > payload
 The contents, as stored.

Enumeration Type Documentation

◆ MetapackEncoding

enum class sourcemeta::core::MetapackEncoding : std::uint8_t
strong

How the contents of a file are stored

Enumerator
Identity 

The contents are stored as they are.

GZIP 

The contents are stored compressed as per RFC 1952.

Function Documentation

◆ metapack_checksum_hex()

SOURCEMETA_CORE_METAPACK_EXPORT auto sourcemeta::core::metapack_checksum_hex ( const MetapackHeader & header) -> std::array< char, METAPACK_CHECKSUM_HEX_LENGTH >

Render the checksum a header records as lowercase hexadecimal. For example:

#include <sourcemeta/core/metapack.h>
#include <sourcemeta/core/io.h>
#include <iostream>
const sourcemeta::core::FileView view{"/tmp/document.metapack"};
const auto parsed{sourcemeta::core::metapack_parse(view).value()};
*parsed.header)};
std::cout.write(checksum.data(), checksum.size());
Definition io_fileview.h:32
SOURCEMETA_CORE_METAPACK_EXPORT auto metapack_parse(const FileView &view) -> std::optional< MetapackView >
SOURCEMETA_CORE_METAPACK_EXPORT auto metapack_checksum_hex(const MetapackHeader &header) -> std::array< char, METAPACK_CHECKSUM_HEX_LENGTH >

◆ metapack_extension_as()

template<typename T>
auto sourcemeta::core::metapack_extension_as ( const std::span< const std::uint8_t > extension) -> const T *

What an extension area holds, as the layout the writer of the file put there, or nothing when it is too small to hold one. For example:

#include <sourcemeta/core/metapack.h>
#include <sourcemeta/core/io.h>
#include <cassert>
#pragma pack(push, 1)
struct Revision {
std::uint32_t number;
};
#pragma pack(pop)
const sourcemeta::core::FileView view{"/tmp/document.metapack"};
const auto parsed{sourcemeta::core::metapack_parse(view).value()};
const auto *revision{
assert(revision != nullptr);
auto metapack_extension_as(const std::span< const std::uint8_t > extension) -> const T *
Definition metapack_read.h:176

◆ metapack_last_modified()

SOURCEMETA_CORE_METAPACK_EXPORT auto sourcemeta::core::metapack_last_modified ( const MetapackHeader & header) -> std::chrono::system_clock::time_point

The point in time a header records. For example:

#include <sourcemeta/core/metapack.h>
#include <sourcemeta/core/io.h>
#include <cassert>
#include <chrono>
const sourcemeta::core::FileView view{"/tmp/document.metapack"};
const auto parsed{sourcemeta::core::metapack_parse(view).value()};
std::chrono::system_clock::now());
SOURCEMETA_CORE_METAPACK_EXPORT auto metapack_last_modified(const MetapackHeader &header) -> std::chrono::system_clock::time_point

◆ metapack_parse()

SOURCEMETA_CORE_METAPACK_EXPORT auto sourcemeta::core::metapack_parse ( const FileView & view) -> std::optional< MetapackView >

Locate every part of a memory-mapped file. For example:

#include <sourcemeta/core/metapack.h>
#include <sourcemeta/core/io.h>
#include <cassert>
const sourcemeta::core::FileView view{"/tmp/document.metapack"};
const auto parsed{sourcemeta::core::metapack_parse(view)};
assert(parsed.has_value());
assert(parsed.value().media_type == "application/json");

◆ metapack_read_json() [1/3]

SOURCEMETA_CORE_METAPACK_EXPORT auto sourcemeta::core::metapack_read_json ( const FileView & view,
const std::uint64_t maximum_decompression_ratio = METAPACK_MAXIMUM_DECOMPRESSION_RATIO ) -> std::optional< JSON >

Read the contents of a memory-mapped file as a JSON document. For example:

#include <sourcemeta/core/metapack.h>
#include <sourcemeta/core/io.h>
#include <cassert>
const sourcemeta::core::FileView view{"/tmp/document.metapack"};
const auto document{sourcemeta::core::metapack_read_json(view)};
assert(document.has_value());
SOURCEMETA_CORE_METAPACK_EXPORT auto metapack_read_json(const std::filesystem::path &path, const std::uint64_t maximum_decompression_ratio=METAPACK_MAXIMUM_DECOMPRESSION_RATIO) -> std::optional< JSON >

◆ metapack_read_json() [2/3]

SOURCEMETA_CORE_METAPACK_EXPORT auto sourcemeta::core::metapack_read_json ( const MetapackView & parsed,
const std::uint64_t maximum_decompression_ratio = METAPACK_MAXIMUM_DECOMPRESSION_RATIO ) -> std::optional< JSON >

Read the contents of an already located file as a JSON document. For example:

#include <sourcemeta/core/metapack.h>
#include <sourcemeta/core/io.h>
#include <cassert>
const sourcemeta::core::FileView view{"/tmp/document.metapack"};
const auto parsed{sourcemeta::core::metapack_parse(view).value()};
const auto document{sourcemeta::core::metapack_read_json(parsed)};
assert(document.has_value());

◆ metapack_read_json() [3/3]

SOURCEMETA_CORE_METAPACK_EXPORT auto sourcemeta::core::metapack_read_json ( const std::filesystem::path & path,
const std::uint64_t maximum_decompression_ratio = METAPACK_MAXIMUM_DECOMPRESSION_RATIO ) -> std::optional< JSON >

Read the contents of a file at the given path as a JSON document. For example:

#include <sourcemeta/core/metapack.h>
#include <cassert>
const auto document{
sourcemeta::core::metapack_read_json("/tmp/document.metapack")};
assert(document.has_value());
assert(document.value().is_object());

◆ metapack_write_bytes()

SOURCEMETA_CORE_METAPACK_EXPORT auto sourcemeta::core::metapack_write_bytes ( const std::filesystem::path & destination,
const std::span< const std::uint8_t > contents,
const std::string_view media_type,
const MetapackEncoding encoding,
const std::span< const std::uint8_t > extension = {} ) -> void

Write bytes as the contents of a file. A media type or extension too large for the format to record throws. For example:

#include <sourcemeta/core/metapack.h>
#include <array>
constexpr std::array<std::uint8_t, 3> bytes{{0x41, 0x42, 0x43}};
"/tmp/blob.metapack", bytes, "application/octet-stream",
SOURCEMETA_CORE_METAPACK_EXPORT auto metapack_write_bytes(const std::filesystem::path &destination, const std::span< const std::uint8_t > contents, const std::string_view media_type, const MetapackEncoding encoding, const std::span< const std::uint8_t > extension={}) -> void
@ Identity
The contents are stored as they are.
Definition metapack_format.h:25

◆ metapack_write_file()

SOURCEMETA_CORE_METAPACK_EXPORT auto sourcemeta::core::metapack_write_file ( const std::filesystem::path & destination,
const std::filesystem::path & source,
const std::string_view media_type,
const MetapackEncoding encoding,
const std::span< const std::uint8_t > extension = {} ) -> void

Write another file as the contents of a file. A media type or extension too large for the format to record throws. For example:

#include <sourcemeta/core/metapack.h>
"/tmp/page.metapack", "/tmp/page.html", "text/html",
SOURCEMETA_CORE_METAPACK_EXPORT auto metapack_write_file(const std::filesystem::path &destination, const std::filesystem::path &source, const std::string_view media_type, const MetapackEncoding encoding, const std::span< const std::uint8_t > extension={}) -> void
@ GZIP
The contents are stored compressed as per RFC 1952.
Definition metapack_format.h:27

◆ metapack_write_json()

SOURCEMETA_CORE_METAPACK_EXPORT auto sourcemeta::core::metapack_write_json ( const std::filesystem::path & destination,
const JSON & document,
const std::string_view media_type,
const MetapackEncoding encoding,
const std::span< const std::uint8_t > extension = {} ) -> void

Write a JSON document as the contents of a file. A media type or extension too large for the format to record throws. For example:

#include <sourcemeta/core/metapack.h>
#include <sourcemeta/core/json.h>
const auto document{sourcemeta::core::parse_json("{ \"foo\": 1 }")};
"/tmp/document.metapack", document, "application/json",
SOURCEMETA_CORE_JSON_EXPORT auto parse_json(std::basic_istream< JSON::Char, JSON::CharTraits > &stream, std::uint32_t maximum_input_bytes=JSON_MAXIMUM_INPUT_BYTES) -> JSON
SOURCEMETA_CORE_METAPACK_EXPORT auto metapack_write_json(const std::filesystem::path &destination, const JSON &document, const std::string_view media_type, const MetapackEncoding encoding, const std::span< const std::uint8_t > extension={}) -> void

◆ metapack_write_json_pretty()

SOURCEMETA_CORE_METAPACK_EXPORT auto sourcemeta::core::metapack_write_json_pretty ( const std::filesystem::path & destination,
const JSON & document,
const std::string_view media_type,
const MetapackEncoding encoding,
const std::span< const std::uint8_t > extension = {} ) -> void

Write a JSON document as the contents of a file, indented for a human to read. A media type or extension too large for the format to record throws. For example:

#include <sourcemeta/core/metapack.h>
#include <sourcemeta/core/json.h>
const auto document{sourcemeta::core::parse_json("{ \"foo\": 1 }")};
"/tmp/document.metapack", document, "application/json",
SOURCEMETA_CORE_METAPACK_EXPORT auto metapack_write_json_pretty(const std::filesystem::path &destination, const JSON &document, const std::string_view media_type, const MetapackEncoding encoding, const std::span< const std::uint8_t > extension={}) -> void

◆ metapack_write_text()

SOURCEMETA_CORE_METAPACK_EXPORT auto sourcemeta::core::metapack_write_text ( const std::filesystem::path & destination,
const std::string_view contents,
const std::string_view media_type,
const MetapackEncoding encoding,
const std::span< const std::uint8_t > extension = {} ) -> void

Write text as the contents of a file, terminated by a line feed. A media type or extension too large for the format to record throws. For example:

#include <sourcemeta/core/metapack.h>
"/tmp/greeting.metapack", "Hello World", "text/plain",
SOURCEMETA_CORE_METAPACK_EXPORT auto metapack_write_text(const std::filesystem::path &destination, const std::string_view contents, const std::string_view media_type, const MetapackEncoding encoding, const std::span< const std::uint8_t > extension={}) -> void

Variable Documentation

◆ METAPACK_CHECKSUM_HEX_LENGTH

std::size_t sourcemeta::core::METAPACK_CHECKSUM_HEX_LENGTH {64}
inlineconstexpr

How many characters it takes to render a checksum

◆ METAPACK_MAXIMUM_DECOMPRESSION_RATIO

std::uint64_t sourcemeta::core::METAPACK_MAXIMUM_DECOMPRESSION_RATIO {1024}
inlineconstexpr

How far stored contents may inflate before a reader refuses them, so that a small file cannot ask for an unbounded amount of memory

◆ METAPACK_MAXIMUM_EXTENSION_BYTES

std::uint32_t sourcemeta::core::METAPACK_MAXIMUM_EXTENSION_BYTES
inlineconstexpr
Initial value:
{
std::numeric_limits<std::uint32_t>::max()}

The largest extension area a file of this format can carry

◆ METAPACK_MAXIMUM_MEDIA_TYPE_BYTES

std::uint16_t sourcemeta::core::METAPACK_MAXIMUM_MEDIA_TYPE_BYTES
inlineconstexpr
Initial value:
{
std::numeric_limits<std::uint16_t>::max()}

The longest media type a file of this format can declare