JSONL

A JSON Lines (https://jsonlines.org) and RFC 7464 JSON text sequence implementation with iterator support. Every non-empty line in a JSONL stream is a complete, valid JSON value of any type, and lines are separated by newline characters (U+000A), optionally preceded by a carriage return (U+000D). Multi-line JSON values are not supported, as per the JSONL specification. More...

Classes
class	sourcemeta::core::JSONL
class	sourcemeta::core::ConstJSONLIterator

Enumerations
enum class	sourcemeta::core::JSONLFraming : std::uint8_t { JSONLFraming::LineFeed , JSONLFraming::RecordSeparator }

Detailed Description

A JSON Lines (https://jsonlines.org) and RFC 7464 JSON text sequence implementation with iterator support. Every non-empty line in a JSONL stream is a complete, valid JSON value of any type, and lines are separated by newline characters (U+000A), optionally preceded by a carriage return (U+000D). Multi-line JSON values are not supported, as per the JSONL specification.

JSON Lines and NDJSON (https://github.com/ndjson/ndjson-spec) describe the same format with minor differences, and this implementation accepts a superset of both:

• Blank and whitespace-only lines are skipped rather than treated as errors. JSON Lines considers them invalid, while NDJSON 3.2 permits ignoring them as long as the behavior is documented
• A newline after the last value is optional. JSON Lines makes it a recommendation, while NDJSON 3.1 requires it when serializing
• Whitespace is tolerated anywhere around a value, including carriage returns that NDJSON 3.1 only allows right before a newline

The same iterator reads RFC 7464 JSON text sequences, the application/json-seq media type, when given the corresponding framing. There, every value is introduced by a record separator (U+001E) rather than terminated by a newline, so a value may span multiple lines. RFC 7464 Section 2.4 requires dropping a top-level number, boolean or null that no whitespace follows, as it may have been truncated, and this implementation honors that. It deviates from the specification as follows:

• Section 2.3 states that a parser "should skip to the next RS" when an element is not a valid JSON text. This implementation throws instead, so that malformed input is never dropped without notice
• Section 2.1 permits ignoring the empty elements that consecutive record separators denote. This implementation also ignores elements that carry nothing but whitespace, matching how it treats blank lines

This functionality is included as follows:

#include <sourcemeta/core/jsonl.h>

---

Class Documentation

sourcemeta::core::JSONL

class sourcemeta::core::JSONL

A range over the JSON documents contained in a JSON Lines stream.

Public Types
enum class	Mode : std::uint8_t { Raw , GZIP }
	The mode of operation for the JSONL parser. More...
Public Member Functions
	JSONL (std::basic_istream< JSON::Char, JSON::CharTraits > &input, Mode mode=Mode::Raw, JSONLFraming framing=JSONLFraming::LineFeed)
	JSONL (const JSONL &)=delete
	JSONL (JSONL &&)=delete

Member Enumeration Documentation

Mode

enum class sourcemeta::core::JSONL::Mode : std::uint8_t

strong

The mode of operation for the JSONL parser.

Enumerator
Raw	The input stream contains raw JSONL text.
GZIP	The input stream contains gzip-compressed JSONL data.

Constructor & Destructor Documentation

JSONL()

sourcemeta::core::JSONL::JSONL	(	std::basic_istream< JSON::Char, JSON::CharTraits > &	input,
		Mode	mode = Mode::Raw,
		JSONLFraming	framing = JSONLFraming::LineFeed )

Parse a JSONL document from a C++ standard input stream using a standard read-only C++ forward iterator interface. An optional mode parameter controls whether the input is treated as raw text or gzip-compressed data, and an optional framing parameter controls how values are delimited. For example, you can parse a JSONL document and prettify each of its rows as follows:

#include <sourcemeta/core/jsonl.h>
#include <cassert>
#include <sstream>
#include <iostream>
 
std::istringstream stream{
  "{ \"foo\": 1 }\n{ \"bar\": 2 }\n{ \"baz\": 3 }"};
 
for (const auto &document : sourcemeta::core::JSONL{stream}) {
  assert(document.is_object());
  sourcemeta::core::prettify(document, std::cout);
  std::cout << '\n';
}

An RFC 7464 JSON text sequence is read the same way, by selecting the record separator framing:

#include <sourcemeta/core/jsonl.h>
#include <cassert>
#include <sstream>
 
std::istringstream stream{"\x1E{ \"foo\": 1 }\n\x1E{ \"bar\": 2 }\n"};
const auto framing{sourcemeta::core::JSONLFraming::RecordSeparator};
 
for (const auto &document :
     sourcemeta::core::JSONL{stream,
                             sourcemeta::core::JSONL::Mode::Raw,
                             framing}) {
  assert(document.is_object());
}

If parsing fails, sourcemeta::core::JSONParseError will be thrown.

sourcemeta::core::ConstJSONLIterator

class sourcemeta::core::ConstJSONLIterator

A forward iterator to parse JSON documents out of a JSON Lines stream. Blank and whitespace-only lines are skipped rather than treated as errors.

Public Member Functions
	ConstJSONLIterator (std::basic_istream< JSON::Char, JSON::CharTraits > *stream, JSONLFraming framing=JSONLFraming::LineFeed)

Constructor & Destructor Documentation

ConstJSONLIterator()

sourcemeta::core::ConstJSONLIterator::ConstJSONLIterator	(	std::basic_istream< JSON::Char, JSON::CharTraits > *	stream,
		JSONLFraming	framing = JSONLFraming::LineFeed )

Construct an iterator over the JSON documents in a stream, optionally selecting the framing that delimits them.

Enumeration Type Documentation

JSONLFraming

enum class sourcemeta::core::JSONLFraming : std::uint8_t

strong

The framing that delimits the JSON values of a stream

Enumerator
LineFeed	Every value is terminated by a line feed, as in JSON Lines and NDJSON.
RecordSeparator	Every value is introduced by a record separator (U+001E) and terminated by a line feed, as in RFC 7464 JSON text sequences