How to Open Large JSON Files

A 400 MB export lands in your downloads folder. You double-click it, your editor greys out, and JSON.parse gives up. Trying to open a large JSON file fails for one specific reason: almost every tool assumes it can hold the entire document in memory before it shows you anything, and past a certain size that assumption stops being true. The fix is not a better viewer. It is reading the file in pieces.

What Actually Runs Out

JSON has no index and no frame markers. You cannot jump to the ten-thousandth record without reading everything in front of it, so a normal parser reads the whole document and builds the whole object graph before returning.

That costs you memory twice. The raw text sits in memory while it is being parsed, and the parsed structure sits next to it. I measured a 30 MB array of 200,000 user objects in Node 22:

So the folklore about JSON needing ten times its size in RAM is wrong for most real data. String-heavy records land near 2x, numeric records near 1x. But you hold the source text at the same time, so plan for roughly two to three times the file size at peak.

Then there is a hard wall. V8 cannot make a string longer than 536,870,888 characters, so reading a file past about 512 MB into a string throws before any parsing starts:

No heap flag fixes that one. Check what heap you have before you start:

You can raise it with node --max-old-space-size=8192 script.js, and that is worth doing for a file in the 200-500 MB range. It buys headroom, not a strategy.

Match the Tool to the Size

VS Code stops rendering a file in the editor past 50 MB, and most GUI JSON viewers fold well before that. Do not spend an afternoon hunting for a desktop app that opens a 2 GB file. That app is jq.

Look Before You Parse

Before anything else, find out what you are dealing with:

The first character tells you most of what matters. A means one giant array, which is the easy case. A { means a top-level object, and you need to know which key holds the bulk. If every line starts with { and ends with }, it is already NDJSON and you can skip straight to line-by-line processing.

Convert to NDJSON Once, Stream Forever

The most useful move with a large JSON array is to turn it into one object per line. Newline-delimited JSON is splittable, greppable, and readable a record at a time by every language.

That command still parses the whole array in memory, so it works up to whatever RAM you have. When it runs out, jq can do the same job without ever holding the full document:

The 1 is the nesting depth to cut at, which is what makes this work on a top-level array. If your records live one level deeper, under a key like results, bump it to 2.

Once the file is NDJSON, ordinary tools work again:

Streaming in Node.js

For an array you cannot fit in memory, stream-json emits records as they are parsed off the read stream:

Memory stays flat because only one record is live at a time. If you already converted to NDJSON, you do not need a library at all:

Streaming in Python

ijson is the equivalent. The path item means each element of the top-level array:

Install it with the C backend (pip install ijson, which ships yajl2_c where available) or the pure-Python fallback will be several times slower on a big file. For records nested under a key, the path becomes results.item.

For NDJSON, plain json.loads per line beats any library:

When You Only Need to Look at It

Half the time the goal is not processing the file, it is understanding its shape so you can write the query or the type definition. Do not open the whole thing. Cut a sample:

Then work with the sample. Paste it into the [JSON viewer to expand the tree and see which keys are nested where, use JSONPath to test the expression you will run against the full file, and run the JSON formatter on any minified chunk you pulled out with head. Browser tools are the right choice for a 50-record slice and the wrong choice for the 400 MB original, so cut first.

If you are shrinking a file for storage rather than reading it, the JSON minifier strips whitespace, though a pretty-printed export usually only loses about a third of its bytes that way. That is not what saves you on a file this size.

The Real Fix Is Upstream

If a file this size lands on you once, stream it and move on. If it arrives every week, the problem is the format, not your parser. Ask whoever produces it for newline-delimited JSON or a paginated endpoint. It costs the producer nothing, and it removes the entire class of problem for everyone downstream.

One gotcha that will cost you an hour otherwise: jq --stream reconstructs values in document order, but if the file is a single top-level object rather than an array, fromstream with the wrong depth silently emits fragments instead of whole records. Test the command on a 50-record sample and diff it against jq -c '.[]' output before you point it at the file that takes twenty minutes to read.