IBM scans 10 billion files in three minutes

Researchers at IBM have succeeded in 43 minutes the gpfs metadata of 10 billion files to read from, among others, by the metadata on flash memory. The previous record by a factor of 37 is broken.

Where IBM in 2007, a record set with the reading of the gpfs metadata of 1 billion files in three hours, it was Big Blue that now with 10 billion files in three quarters, as the company has announced. Gpfs, in full General Parallel File System, which is IBM’s file system for clustercomputing. One file is in use by gpfs at multiple physical locations; quickly able to interpret the metadata is crucial for the performance.

The record was achieved with a cluster of ten IBM xSeries servers each with a dual-quadcore-cpu and 32GB of memory; the metadata were placed on four solid-state arrays from Violin Memory. The data were in total 6.5 terabytes large. The Violin arrays were all pci-express to two IBM servers linked, which in turn via infiniband through a 24 ports switch SilverStorm were knotted.

The broken record were the data on a disk is placed; also by the metadata on flash memory was this record achieved. In addition, the gpfs-algorithms since 2007 improved.