Parallel Data Mover

Parallel Data Transfer: Maximum Throughput for Ultra-scalable Filing Systems like Lustre

The ideal addition for high-performance systems: The Parallel Data Mover (PDM) allows unlimited data transfer

between ultra-scalable filing systems and backend technologies. Maximize your goodput!

Whether it be the industry, research or a governmental institution – if you process large quantities of data, you need powerful technologies. The PDM is the ideal solution for this.

Having been developed especially to meet demands of high-performance systems, it allows maximum goodput between parallel filing systems and POSIX-based backend technologies. This opens up new possibilities for processing data.

PDM can help you lower your expenses for data storage by building an HSM archiving system. The data will be swapped and stored on cheaper storage devices instead of the costly high-performance system. Combined with the open source program OpenArchive, this is an affordable option for long-term archiving of large amounts of data.

The PDM allows you to rely on a simple and unimpeded application.

If nodes or certain infrastructures should become inoperative, the PDM recognizes it and automatically transfers the workload to the remaining nodes – without disrupting the running operation. In most cases, you won’t even notice the outage. In addition to that, administration is kept simple: Each PDM entity operates within a configured subnetwork and controls the workloads and the necessary resources in these subnets automatically.

The PDM is independent in terms of hardware and not tied to certain filing systems. This way, it can be easily incorporated into your IT system. Secure unlimited data transfer for your system!

PROBLEM DESCRIPTION

Parallel filing systems such as Lustre are ideal for processing huge amounts of data extremely fast. However, it can be very expensive and grossly inefficient to store data there for a long time. At the same time, it is difficult to copy the enormous amounts of data from the parallel filing system to the backend. How can data be swapped without losing the instant availability?

The solution is the Parallel Data Mover (PDM). Parallel data swapping maximizes the capacity to copy from parallel filing systems to backend technologies. This enables efficient HSM archives and extremely fast data transfers.

FUNCTIONALITY

The PDM is the ideal solution for copying large amounts of data. The software’s special mechanism allows copying several files at the same time. Files are being bundled into workloads and transferred to workers which are distributed between the various nodes. The optimized design makes it possible to reach a copy capacity of up to 5 GB/s per node or 1 TB/s with 200 nodes. Still not enough? The PDM is scalable up to several thousand nodes – according to the individual demands of your systems.

More nodes can be added easily anytime.

The Parallel Data Mover offers you flexibility to use different filing systems and supports all standard POSIX filing systems, Lustre filing systems, as well as PLFS. Data transfer is made via Lustre HSM API or REST API.

Do you want to maximize your data transfern? Do you have any questions concerning the Parallel Data Mover or do you require individual advice for your company? Feel free to directly contact us here.