In this post we open up a small Parquet file and look at how it’s laid out. Along the way we’ll see why a reader can often skip most of the file. To keep things simple, the file has no compression or encryption and uses only plain encoding. Later posts cover those.
1Row groups and column chunks
Take an example table with 8 rows and 3 columns: id, name and age. id and age are whole numbers (32-bit integers) and name is a string.1 A row-oriented format such as CSV stores it one row after another: 1, ann, 34, then 2, bob, 27, and so on. A columnar format stores all of id, then all of name, then all of age, so a query that needs only age reads only those bytes.
Parquet does a bit of both. Rows are collected into row groups, here 2 groups of 4 rows.2 Within a row group, each column's values are stored together as a column chunk. The file holds the chunks one after another: id, name and age for row group 0, then the same for row group 1. This layout is known as PAX.3
2The whole file: data, then a footer
PAR1, which mark it as Parquet.4 After the data comes the footer, which describes the data: its columns, and where each column chunk starts.5 Just before the final PAR1, 4 bytes give the footer's length.3Pages: a chunk in smaller pieces
A column chunk can hold millions of values, so it is split into pages: smaller pieces that are encoded, compressed and read one at a time. So a reader can hand over a chunk's first values before it has decoded the rest.6
Pages are also where Parquet makes data smaller. A page's values can be encoded, for example with a dictionary that stores each distinct value once and then small numbers pointing into it, and then compressed with a codec such as snappy or zstd. Both work well because a page holds one column's values, which tend to be alike. The example file uses neither: its values are written as they are (PLAIN) and uncompressed, so every byte stays readable.
Here each of the 6 chunks holds 2 pages, and each page holds 2 values. Each page starts with a header, 17 bytes in the example file, then its values. Values can vary in size, since a string can be any length, so the header records how many bytes of values follow: that's how a reader knows where the page ends.7
A 32-bit integer, such as an id or age, takes 4 bytes. A string, such as a name, takes 4 bytes for its length, then its bytes: 4 + 3 bytes for ann. So a page of name holds 14 bytes of values, and a page of id or age holds 8.
page headerlength of a name, 4 bytesits bytes, 3 for ann
4The bytes, one cell each
00 to ff, and coloured by the part it belongs to. The number at the start of a row is its offset: how many bytes into the file the row starts. On the right, each byte that is a printable character is shown as one, so the values of name and the column names stand out as text. Most of the bytes are numbers in a binary form that can't be read by eye, but a few can. Those are listed next to the dump.6Reading from the end
A reader starts with the footer, because that's where it learns where each column chunk is. The footer is at the end of the file, so a reader begins there.9
It reads the last 8 bytes, the footer's length and PAR1, then the 351 bytes before them: the footer.10 For SELECT name it then fetches only the 2 name chunks, 124 of the 324 bytes of data. The bytes of id and age are never read.
SELECT name.7Inside a chunk, page by page
The footer says the first name chunk starts at byte 54 and is 62 bytes long, but it says nothing about the pages inside it. The reader finds them one at a time.
At byte 54 it reads a 17-byte page header. The header says how many values the page holds and how many bytes they take: 2 values in 14 bytes. So the values are bytes 71 to 85, ann and bob, and the next header starts right after them, at 85. After the second page the reader is at byte 116. That's 54 + 62, the end of the chunk, so it stops.11
name chunk, page by page.8How the example file was written
| setting | example file | arrow-rs default |
|---|---|---|
| most rows per row group | 4 | 1,048,576 |
| most rows per page | 2 | 20,000 |
| rows handed to the writer at a time | 2 | 1,024 |
| compression | none | none |
| dictionary encoding | off | on |
| min / max statistics | off | per chunk and per page |
| page index | off | on |
| a copy of the Arrow schema in the footer | off | on |