· part 1 of Parquet explorations

Inside a Parquet file

In this post we open up a small Parquet file and look at how it’s laid out. Along the way we’ll see why a reader can often skip most of the file. To keep things simple, the file has no compression or encryption and uses only plain encoding. Later posts cover those.

1Row groups and column chunks

Take an example table with 8 rows and 3 columns: id, name and age. id and age are whole numbers (32-bit integers) and name is a string.1 A row-oriented format such as CSV stores it one row after another: 1, ann, 34, then 2, bob, 27, and so on. A columnar format stores all of id, then all of name, then all of age, so a query that needs only age reads only those bytes.

Parquet does a bit of both. Rows are collected into row groups, here 2 groups of 4 rows.2 Within a row group, each column's values are stored together as a column chunk. The file holds the chunks one after another: id, name and age for row group 0, then the same for row group 1. This layout is known as PAX.3

idnameagerow group 0row group 0idnameagerow group 1row group 1idnameage111annannann343434222bobbobbob272727333catcatcat454545444dandandan191919555eveeveeve525252666fayfayfay383838777gusgusgus616161888halhalhal232323row group 0 · idrow group 0 · namerow group 0 · agerow group 1 · idrow group 1 · namerow group 1 · age1ann342bob273cat454dan195eve526fay387gus618hal23

2The whole file: data, then a footer

A Parquet file starts and ends with the 4 letters PAR1, which mark it as Parquet.4 After the data comes the footer, which describes the data: its columns, and where each column chunk starts.5 Just before the final PAR1, 4 bytes give the footer's length.
data · 324 Bfooter · 351 B328PAR15f010000PAR104679683687= 351 (read lowest byte first),the footer's length

3Pages: a chunk in smaller pieces

A column chunk can hold millions of values, so it is split into pages: smaller pieces that are encoded, compressed and read one at a time. So a reader can hand over a chunk's first values before it has decoded the rest.6

Pages are also where Parquet makes data smaller. A page's values can be encoded, for example with a dictionary that stores each distinct value once and then small numbers pointing into it, and then compressed with a codec such as snappy or zstd. Both work well because a page holds one column's values, which tend to be alike. The example file uses neither: its values are written as they are (PLAIN) and uncompressed, so every byte stays readable.

Here each of the 6 chunks holds 2 pages, and each page holds 2 values. Each page starts with a header, 17 bytes in the example file, then its values. Values can vary in size, since a string can be any length, so the header records how many bytes of values follow: that's how a reader knows where the page ends.7

A 32-bit integer, such as an id or age, takes 4 bytes. A string, such as a name, takes 4 bytes for its length, then its bytes: 4 + 3 bytes for ann. So a page of name holds 14 bytes of values, and a page of id or age holds 8.

row group 04 rowsid 50 B · 4–54page 0header1 2page 1header3 4name 62 B · 54–116page 0headerann bobpage 1headercat danage 50 B · 116–166page 0header34 27page 1header45 19row group 14 rowsid 50 B · 166–216page 0header5 6page 1header7 8name 62 B · 216–278page 0headereve faypage 1headergus halage 50 B · 278–328page 0header52 38page 1header61 23

page headerlength of a name, 4 bytesits bytes, 3 for ann

4The bytes, one cell each

Here are all 687 bytes. Each cell is one byte, written as two hex digits from 00 to ff, and coloured by the part it belongs to. The number at the start of a row is its offset: how many bytes into the file the row starts. On the right, each byte that is a printable character is shown as one, so the values of name and the column names stand out as text. Most of the bytes are numbers in a binary form that can't be read by eye, but a few can. Those are listed next to the dump.
016324864809611212814416017619220822424025627228830432033635236838440041643244846448049651252854456057659260862464065667250P41A52R31115·00·15·10·15·10·2c,15·04·15·00·15·06·15·06·00·00·01·00·00·00·02·00·00·00·15·00·15·10·15·10·2c,15·04·15·00·15·06·15·06·00·00·03·00·00·00·04·00·00·00·15·00·15·1c·15·1c·2c,15·04·15·00·15·06·15·06·00·00·03·00·00·00·61a6en6en03·00·00·00·62b6fo62b15·00·15·1c·15·1c·2c,15·04·15·00·15·06·15·06·00·00·03·00·00·00·63c61a74t03·00·00·00·64d61a6en15·00·15·10·15·10·2c,15·04·15·00·15·06·15·06·00·00·22"00·00·00·1b·00·00·00·15·00·15·10·15·10·2c,15·04·15·00·15·06·15·06·00·00·2d-00·00·00·13·00·00·00·15·00·15·10·15·10·2c,15·04·15·00·15·06·15·06·00·00·05·00·00·00·06·00·00·00·15·00·15·10·15·10·2c,15·04·15·00·15·06·15·06·00·00·07·00·00·00·08·00·00·00·15·00·15·1c·15·1c·2c,15·04·15·00·15·06·15·06·00·00·03·00·00·00·65e76v65e03·00·00·00·66f61a79y15·00·15·1c·15·1c·2c,15·04·15·00·15·06·15·06·00·00·03·00·00·00·67g75u73s03·00·00·00·68h61a6cl15·00·15·10·15·10·2c,15·04·15·00·15·06·15·06·00·00·34400·00·00·26&00·00·00·15·00·15·10·15·10·2c,15·04·15·00·15·06·15·06·00·00·3d=00·00·00·17·00·00·00·15·02·19·4cL48H0c·61a72r72r6fo77w5f_73s63c68h65e6dm61a15·06·00·15·02·25%00·18·02·69i64d00·15·0c·25%00·18·04·6en61a6dm65e25%00·4cL1c·00·00·00·15·02·25%00·18·03·61a67g65e00·16·10·19·2c,19·3c<26&00·1c·15·02·19·25%00·06·19·18·02·69i64d15·00·16·08·16·64d16·64d26&08·49I1c·15·00·15·00·15·04·00·00·00·26&00·1c·15·0c·19·25%00·06·19·18·04·6en61a6dm65e15·00·16·08·16·7c|16·7c|26&6cl49I1c·15·00·15·00·15·04·00·00·00·26&00·1c·15·02·19·25%00·06·19·18·03·61a67g65e15·00·16·08·16·64d16·64d26&e8·01·49I1c·15·00·15·00·15·04·00·00·00·16·c4·02·16·08·26&08·16·c4·02·14·00·00·19·3c<26&00·1c·15·02·19·25%00·06·19·18·02·69i64d15·00·16·08·16·64d16·64d26&cc·02·49I1c·15·00·15·00·15·04·00·00·00·26&00·1c·15·0c·19·25%00·06·19·18·04·6en61a6dm65e15·00·16·08·16·7c|16·7c|26&b0·03·49I1c·15·00·15·00·15·04·00·00·00·26&00·1c·15·02·19·25%00·06·19·18·03·61a67g65e15·00·16·08·16·64d16·64d26&ac·04·49I1c·15·00·15·00·15·04·00·00·00·16·c4·02·16·08·26&cc·02·16·c4·02·14·02·00·28(19·70p61a72r71q75u65e74t2d-72r73s20 76v65e72r73s69i6fo6en20 3553882e.3332e.30019·3c<1c·00·00·1c·00·00·1c·00·00·00·5f_01·00·00·50P41A52R311

6Reading from the end

A reader starts with the footer, because that's where it learns where each column chunk is. The footer is at the end of the file, so a reader begins there.9

It reads the last 8 bytes, the footer's length and PAR1, then the 351 bytes before them: the footer.10 For SELECT name it then fetches only the 2 name chunks, 124 of the 324 bytes of data. The bytes of id and age are never read.

The reads for SELECT name.
ididfooterrow group 0row group 16871last 8 bytes: footer length and PAR12footer, 351 B3name, row group 0, 62 B4name, row group 1, 62 B4 reads, 483 of 687 bytes

7Inside a chunk, page by page

The footer says the first name chunk starts at byte 54 and is 62 bytes long, but it says nothing about the pages inside it. The reader finds them one at a time.

At byte 54 it reads a 17-byte page header. The header says how many values the page holds and how many bytes they take: 2 values in 14 bytes. So the values are bytes 71 to 85, ann and bob, and the next header starts right after them, at 85. After the second page the reader is at byte 116. That's 54 + 62, the end of the chunk, so it stops.11

The first name chunk, page by page.
headerann bobheadercat danheader, 17 Bvalues, 14 Bheader, 17 Bvalues, 14 B547185102116at 116 the chunk's 62 bytes are done: it held 2 pages

8How the example file was written

We wrote the example file with arrow-rs and changed a few of its settings to keep it small enough to read byte by byte. With the default settings, the same 8 rows would come out differently. They would be one row group, and each column chunk would start with a dictionary page. The footer would also hold each column's smallest and largest values and a copy of the Arrow schema, and a page index would follow the data.12 The table shows each setting.13
settingexample filearrow-rs default
most rows per row group41,048,576
most rows per page220,000
rows handed to the writer at a time21,024
compressionnonenone
dictionary encodingoffon
min / max statisticsoffper chunk and per page
page indexoffon
a copy of the Arrow schema in the footeroffon