0

Limit NDJSON Line Size Before Parsing an Inventory Record

Reading a whole file before checking record size defeats the purpose of a size limit. For a binary NDJSON stream, read at most one byte beyond the permitted line size and reject an oversized line before parsing it.

An inventory export used in global dropshipping can contain one JSON object per line. This example provides a small parsing boundary for warehousing and fulfillment tooling, with no customer data or live API.

import io
import json

def reject_constant(value):
    raise ValueError("non-finite JSON number")

def iter_records(stream, max_line_bytes):
    if type(max_line_bytes) is not int or max_line_bytes < 2:
        raise ValueError("invalid line limit")
    while True:
        raw = stream.readline(max_line_bytes + 1)
        if raw == b"":
            return
        if len(raw) > max_line_bytes:
            raise ValueError("line too large")
        if not raw.strip():
            continue
        record = json.loads(raw.decode("utf-8"),
                            parse_constant=reject_constant)
        if not isinstance(record, dict):
            raise ValueError("object required")
        yield record

source = io.BytesIO(b'{"sku":"0007"}\n{"sku":"A-2"}\n')
print(list(iter_records(source, 64)))

The result contains two dictionaries in input order. The limit includes the line terminator, so CRLF consumes two bytes. A final record without a newline is accepted when it fits. The input contract is a binary stream; this helper is not a text-stream adapter.

A streaming limit is not an atomic import

The generator can yield a valid record before discovering an error on the next line. It cannot undo work already performed by its caller. If the application requires an all-or-nothing database import, that transaction boundary must be designed separately.

The exact code was run against LF and CRLF input, a UTF-8 value, a final line without a newline and blank lines. Checks also covered an oversized line, malformed JSON, invalid UTF-8, a top-level array, non-finite numbers and invalid limit values. A prefix-yield test confirmed the partial-progress behavior described above.

This limit bounds the bytes requested for one line; it does not limit the total number of records or establish a complete inventory schema. Python's JSON documentation explains decoding and the parse_constant hook. Validate business fields after parsing, and decide how a rejected file affects previously processed records before connecting this boundary to warehouse work.

Computer workstations and printing equipment in a warehouse.

Computer workstations and printing equipment in a warehouse.


All rights reserved

Viblo
Hãy đăng ký một tài khoản Viblo để nhận được nhiều bài viết thú vị hơn.
Đăng kí