Enterprise AI Analysis
From Edge to HPC: Investigating Cross-Facility Data Streaming Architectures
This paper explores three cross-facility data streaming architectures—Direct Streaming (DTS), Proxied Streaming (PRS), and Managed Service Streaming (MSS)—evaluating their performance and deployment feasibility using the Data Streaming to HPC (DS2HPC) framework on the OLCF's Advanced Computing Ecosystem. Our findings show that DTS offers the highest throughput and lowest latency due to its minimal-hop path, while MSS provides greater deployment feasibility and scalability but incurs significant overhead. PRS strikes a balance, offering scalability with performance comparable to DTS in many scenarios. This research provides critical insights for integrating experimental facilities with HPC systems for near real-time data analysis and AI-driven decision-making.
Executive Impact: Streamlining Cross-Facility Data Flows
Our analysis reveals key performance indicators and architectural trade-offs essential for high-performance computing and AI integration, driving real-time scientific discovery.
0
Max DTS Throughput
0
Min DTS Latency (RTT)
0
MSS Overhead vs. DTS
Deep Analysis & Enterprise Applications
Select a topic to dive deeper, then explore the specific findings from the research, rebuilt as interactive, enterprise-focused modules.
Architectures
Performance Insights
Real-World Workloads
Enterprise Process Flow
Direct Streaming (DTS)
→
Proxied Streaming (PRS)
→
Managed Service Streaming (MSS)
| Feature | Direct Streaming (DTS) | Proxied Streaming (PRS) | Managed Service Streaming (MSS) |
|---|---|---|---|
| Data Flow Path | Minimal-hop, direct node-level access. | Intermediary proxies, logical overlay. | Facility-managed services, FQDN-based access. |
| Deployment Complexity | Simplest, but manual firewall/iptables config required. | Moderate, proxy/tunnel setup adds complexity. | Most streamlined, managed APIs. |
| Security & Network | Exposes node-level access, bypasses standard policies. | Encapsulates traffic with mTLS, stable endpoints. | Handles routing, DNS, security enforcement. |
| Scalability | Poor (manual port assignment). | Scalable (via proxies). | Scalable (load balancer, managed services). |
39K
Messages/sec with Direct Streaming (Work Sharing)
Direct Streaming (DTS) delivers superior throughput, achieving up to 39K messages/sec in work sharing scenarios with minimal latency, critical for real-time analytics. Proxied Streaming (PRS), especially with HAProxy, closely matches DTS performance, reaching up to 19K messages/sec and maintaining tight RTT distributions. Conversely, Managed Service Streaming (MSS), while offering simplified deployment, incurs significant overhead, particularly in feedback-intensive workloads, with RTTs spiking to 40 seconds at higher consumer counts for Lstream (larger payload) workloads.
Our evaluation indicates that for the work-sharing with feedback pattern, PRS often performs as well as or better than DTS. However, MSS consistently shows greater overhead (up to 6.9x for Dstream workloads) due to its layered architecture and load balancing mechanisms. These insights are vital for optimizing cross-facility data pipelines where latency and throughput are paramount.
Case Study: LCLS & Deleria Workflows
The Linac Coherent Light Source (LCLS) workflow at SLAC National Accelerator Laboratory streams diffraction frames at up to 100 GB/s to OLCF HPC systems. Here, AI models perform real-time analysis to identify Bragg peaks and guide experimental parameter changes while the sample is still in the beam.
Similarly, the GRETA/Deleria workflow at Michigan State University streams 500K events/sec (32 Gbps) for real-time gamma-ray energy and 3D position analysis. These critical scientific workflows demonstrate the imperative for efficient, low-latency, cross-facility data streaming to enable AI-driven scientific discovery and real-time experimental steering.
The study utilized three messaging patterns: work sharing, work sharing with feedback, and broadcast and gather, aligning with common AI-HPC communication motifs. The Dstream workload (Deleria-based) involves KiB-range messages with variable event counts, sustaining up to 32 Gbps. The Lstream workload (LCLS-based) uses 1 MiB HDF5-formatted payloads, streaming at up to 30 Gbps, essential for responsive analysis and experiment steering, including AI model retraining.
Calculate Your Potential AI ROI
Estimate the productivity gains and cost savings from implementing optimized AI data streaming architectures in your enterprise.
Your AI Implementation Roadmap
A phased approach to integrate cross-facility data streaming and AI into your enterprise, ensuring robust and scalable solutions.
Phase 1: Discovery & Strategy
Conduct a detailed assessment of existing data infrastructure and workflow requirements. Define clear AI integration goals and select optimal streaming architectures (DTS, PRS, MSS) based on performance and security needs.
Phase 2: Pilot & Proof-of-Concept
Implement a small-scale pilot project leveraging the DS2HPC framework and SciStream toolkit. Validate data flow paths, measure streaming throughput and latency, and refine configurations for chosen workloads (e.g., LCLS, Deleria).
Phase 3: Scalable Deployment
Roll out the selected streaming architecture across your enterprise. Optimize for large-scale data streams and integrate with HPC systems for advanced AI model training and real-time inference. Establish monitoring and feedback loops for continuous improvement.
Phase 4: Operational Excellence
Transition to full operational status with ongoing support and maintenance. Explore advanced features like dynamic compute orchestration and robust security protocols. Continuously adapt to evolving scientific workflows and technological advancements.
Ready to Transform Your Data Strategy?
Book a free consultation with our AI streaming experts to explore how these architectures can revolutionize your enterprise's data-driven initiatives.
Ready to Get Started?
Book Your Free Consultation.
Let's Discuss Your AI Strategy!
Lets Discuss Your Needs
Select a Date
Sun
Mon
Tue
Wed
Thu
Fri
Sat