NAS, S3, Disk - wherever files live today. No migration required.
Solution Design Guide
Intelligent Tape Archive.
A design guide for building archives that capture intelligence before data goes to cold storage - keeping insights accessible for AI, analytics, and compliance, without the egress costs.
01 / Context
Why intelligent archiving matters.
Traditional archives become black boxes. Data goes in, and the insights stay locked away - unreachable by the models, analysts, and auditors who need them most.
The problem today
- AI can't see archived data
- Egress costs block access to insights
- No visibility into what you actually have
- Compliance requires expensive recalls
- Data hoarded on costly primary storage
The opportunity
- Capture intelligence before archive
- Query insights without moving data
- Complete transparency into cold storage
- AI and compliance workflows enabled
- Archive faster and reduce storage costs
The Goal
An archive where intelligence stays accessible - even after the files themselves go cold.
03 / Architecture Pattern
Intelligence layer + deep archive.
Three concerns, cleanly separated. Active storage stays fast. The intelligence layer stays queryable. The deep archive stays cheap.
03a / The Two Layers
MetadataHub + XtreemStore.
The intelligence layer and the deep archive layer - purpose-built, independently scalable, and designed to work together.
MetadataHub
The Intelligence Layer.
Always-hot proxy for files.
- Extracts context, insights, and deep metadata
- Persists a queryable index across all storage
- Acts as the always-hot proxy for files
- Answers "What's in my files and on tape?"
We make data findable.
XtreemStore
The Deep Archive Layer.
Files at rest on tape.
- S3-compatible tape object storage
- Scalable, low-cost cold tier
- Files at rest on tape
- Hardware agnostic, no vendor lock-in
Infinitely scalable and affordable.
Together, tape becomes an active AI tier.
Tape stores the data. MetadataHub stores the intelligence. XtreemStore makes the archive infinitely scalable and affordable. The intelligence layer stays hot while files stay cold - active workflows, cold-storage economics.
03b / Working Together
How MdH + XtreemStore work together.
A single flow, four stages. Intelligence is captured once, then queried forever - while files move automatically to the cheapest tier.
1. Source Storage
2. MetadataHub harvests and indexes intelligence - once.
Rich metadata, structure, relationships, and context captured at ingest. Build once. Query forever.
3. Policy-driven tiering
Automatic tiering and migration via your data-mover of choice. Files move from source to XtreemStore based on policy - no manual handoff, no lost context.
Data-mover partners
- Panzura Symphony
- Starfish
- Mediaflux
4. Deep Archive on XtreemStore
Files at rest on tape. Intelligence stays always-online via MetadataHub - queryable without recall.
Result
90%+ of AI, analytics, and compliance queries answered from the intelligence layer. Files stay cold until truly needed - zero egress for discovery.
04 / Outcomes
What this enables.
The same intelligence layer unlocks three workloads that traditional archives simply cannot support.
AI workflows
Point your models straight at the intelligence layer. No recalls, no waiting, no egress bill just to find and feed the right data to AI.
Compliance
Answer audits from metadata and context. Retrieve files only when they are truly required by the regulator.
Cost reduction
Archive aggressively with full visibility. Most operations never touch the cold tier - so they never pay the egress bill.
90% +
Queries answered without egress
1 / 1000
Metadata proxy size vs. original
$0
Egress cost for intelligence queries
05 / Implementation and Best Practices
Implementation considerations.
How to turn these principles into production reality - and the habits that separate successful intelligent archives from failed ones.
1. Intelligence Extraction
- Extract rich embedded metadata, structure, relationships and context
- Index once at ingest or at first access time
- Build once, query forever
- Schema-on-read for evolving attribute sets
2. Storage Architecture
- S3-compatible interface for the deep archive tier
- Scale intelligence and archive layers independently
- Files written to S3 - tape or cloud, your choice
- Intelligence remains always accessible
3. Data Organization
- Group related files for efficient batch retrieval
- Tag-based routing to containers
- Containers span multiple tapes - no single-tape size limits
- Retention and legal holds at the container level
4. Query and Access
- Global search across all archived data
- Filter by any captured attribute
- Retrieve only what you actually need
- Feed AI and analytics directly from the intelligence layer
Results
Measured Impact
Results from production deployments.
40 - 70 %
Reduction in redundant processing
1,000x
Fewer archive recalls
<6 months
Typical payback period
Customer Stories
Real results from real deployments.
Eliminating the AI Token Tax on 200 Petabytes in Research
How one of Europe's largest scientific data centers stopped reprocessing the same files.
"At Zuse Institute Berlin, managing 200 petabytes meant our scientists repeatedly reopened and reprocessed the same files. MetadataHub changed that - we could finally access the information instantly without touching the underlying storage."
Carsten Schaeuble
Head of Group, IT & Data Services, Zuse Institute Berlin
PB-scale
archive now fully searchable
From Petabyte Blindness to Instant AI-Ready Microscopy
MetadataHub transformed years of microscopy research data into instantly searchable, AI-ready assets.
"In seconds I can find 80,000 microscopy images at exactly 300 nm resolution - something that was impossible before MetadataHub."
Dr. Yannic Kerkoff
Researcher, Zuse Institute Berlin
PB-scale
microscopy data now searchable