The fst package for R provides a fast, easy and flexible way to serialize data frames. With access speeds of multiple GB/s, fst is specifically designed to unlock the potential of high speed solid state disks that can be found in most modern computers. Data frames stored in the fst format have full random access, both in column and rows.
The figure below compares the read and write performance of the fst package to various alternatives.
These benchmarks were performed on a laptop (i7 4710HQ @2.5 GHz) with a reasonably fast SSD (M.2 Samsung SM951) using the dataset defined below. Parameter Speed was calculated by dividing the in-memory size of the data frame by the measured time. These results are also visualized in the following graph:
LZ4 - Extremely fast compression
Zstandard - Real-time data compression algorithm
Bob's Blog – Finance, programming and stuff
Zstandard - Real-time data compression algorithm
Larger-than-RAM Disk-Based Data Manipulation Framework • disk.frame
Economics and R
Bob's Blog – Finance, programming and stuff