ANTOR: A Secure Peer-to-Peer File Transfer Protocol for the ANNE Network

A BitTorrent-inspired, privacy-preserving file transfer protocol for reliable chunked file distribution across ANNODEs

I. Introduction to ANTOR

In contrast to traditional file transfer protocols that often rely on central coordination points and may inadvertently expose sensitive metadata, ANTOR operates independently within a fully decentralized network of nodes. This enables secure transfer of a diverse array of file types, including documents, media, and application data, between participants leveraging personal or edge computing hardware.

ANTOR is activated in scenarios involving the retrieval of content from remote nodes or sharing data across the network. It is versatile enough to handle both openly accessible materials and content restricted to authorized recipients, all while imposing minimal strain on the underlying consensus layer. This design paradigm accommodates a wide spectrum of applications, from collaborative knowledge databases to secure data replication, all within a framework prioritizing user autonomy and operational resilience.

Core characteristics include:

• Confidential Transmission: All interactions are encrypted, preventing exposure of content or metadata.
• Independent Verification: Each segment is individually hashed, ensuring integrity throughout the transfer process.
• Fault Tolerance: Retrieval from multiple sources with built-in mechanisms for handling failures and retries.
• Resource Efficiency: On-demand loading and temporary buffering accommodate constrained environments.
• System Integration: Tight coupling with file metadata handling, relational knowledge structures, and local node operation.

II. Core Components and Data Model

II.I. Transfer Segments (Ants)

Content is partitioned into uniform segments of fixed size, configurable per file transfer; the default size is 512,000 bytes (512 decimal kilobytes). Each segment, termed an ant, is encoded into a text-safe base64 format for transmission and assigned a unique cryptographic digest for validation. Ants are stored locally in structured directories based on content classification: a2a_neurons for shared content, a2a_cache for temporary data, and a2a_localonly for private files, allowing organized access and clear separation of concerns.

Segment state progresses through distinct phases defined by enumeration: not started, pending, failed, done, sent, or preloaded. Tracking records maintain attempt history (peer identifiers and timestamps), timing information, and diagnostic notes, enabling adaptive retry behavior based on elapsed time and past peer reliability.

II.II. Manifest Records (ANTOR Files)

Every transfer is governed by a manifest record, stored as a JSON file with the extension .antor. This record captures the complete structure of the file. It includes the total size, encoded length, overall digest, protocol version, and an ordered list of segment entries, each specifying position and corresponding hash. An optional providers array may list known custodians (by node identifier) to aid discovery and sourcing.

Manifests are created during initial preparation (by the provider) or derived from received information (by the requestor). They serve as authoritative references for both distribution and reassembly, bridging fragmented transmission with unified restoration.

III. Operational Dynamics: Distribution and Retrieval

III.I. Distribution Role: Content Provision

When a request arrives for file metadata, the custodian (provider) verifies local possession and recipient eligibility. For content restricted to local-only access, a per-peer approval list is consulted. If conditions are satisfied, the manifest is prepared or retrieved and returned through an encrypted channel, including structural details, size, hash, and segment inventory. Subsequent requests for individual ants trigger loading from storage, validation of integrity via the stored hash, and encrypted delivery, after which resources are released.

III.II Retrieval Role: Coordinated Acquisition

The requesting node (requestor) begins by estimating the expected segment count from preliminary size information. Requests for metadata are sent to candidate contributors over encrypted channels. Upon receiving the manifest, the segment list is populated and parallel retrieval begins across available sources using a configurable thread pool (default maximum of four concurrent operations).

Each arriving ant is validated against its recorded hash, integrated into local storage, and cleared from temporary buffers. Failures trigger reassessment and redirection to alternative sources, with automatic retries after a timeout interval. Once all ants are obtained, the file is reconstructed by sequential concatenation of base64 strings, decoded to its original binary form, validated against the overall digest, and stored in the appropriate repository (neurons, cache, or local-only). Completion is signaled through system events, and transient records are eventually removed after an expiration period (default 300 seconds).

IV. Protective Frameworks: Confidentiality and Integrity

ANTOR incorporates multiple layers of protection:

• Encrypted Communication: All message exchanges use authenticated encryption based on a shared secret derived from the sender’s private key and the recipient’s public key (via elliptic-curve Diffie-Hellman). This ensures both confidentiality and that the message originated from the claimed sender.
• Access Control: Restricted content (local-only) requires explicit per-participant authorization before any disclosure; an internal approval map tracks permitted peers.
• Integrity Assurance: Independent cryptographic digests (MD5 of base64 data) on each ant and the complete payload detect any unauthorized modification; mismatches trigger failure and redirection.
• Selective Visibility: Custodians disclose content existence and structure only to pre-authorized peers; for local-only files, the manifest is withheld unless the peer is approved.
• Decentralized Discovery: No central registry is required; location relies on ambient network awareness (active peer list) and relational references stored in the hypergraph (via file metadata relons).

V. Performance and Implementation Characteristics

ANTOR is designed for deployment on resource-constrained devices, including consumer hardware running ANNODEs:

• Concurrency Control: Limited parallel operations to a default 4 prevent overload while maximizing throughput; a dedicated executor service manages ant retrieval tasks.
• Memory Optimization: Ant data is loaded from disk only when needed (on provider side) and released immediately after transmission; on the requestor side, incoming ant data is persisted to disk promptly to minimize heap usage.
• Lifecycle Management: Resolved transfers are automatically removed from memory after a short retention period (default 300 seconds), with periodic cleanup routines.
• Storage Organization: Segregated directories (a2a_neurons, a2a_cache, a2a_localonly) maintain separation between shared, cached, and private content, with file paths derived from the file’s numeric identifier.

By utilizing existing peer communication channels, the file transfer protocol adds negligible overhead while inheriting the network’s established trust and encryption capabilities.

VI. Integration with the ANNE Ecosystem

ANTOR operates in close coordination with other ANNE layers:

• Neuromorphic Hypergraph: File metadata, including content hashes and references to manifests, are stored as relons, enabling semantic discovery and contextual linkage within the knowledge graph.
• Alt Data Network: ANTOR provides the transfer mechanism for unstructured payloads indexed by the alt data layer, bridging the gap between metadata and actual content.
• ANNODE Local Storage: Files are served directly from the node’s file system, with access controlled by key-based permissions and directory segregation.
• Peer Messaging Layer: All ANTOR communications use the existing encrypted peer channels, avoiding separate connection overhead.

This integration guarantees that files transition from simple transfers to becoming fully addressable, discoverable, and verifiable elements within the broader knowledge graph.

VII. Theoretical Foundations and Systemic Properties

As a purely peer-to-peer file transfer protocol, ANTOR synthesizes established concepts from distributed systems: parallel segmentation for scalability, hash-based verification for tamper resistance, and distributed coordination for robustness. The protocol implements a content-addressable overlay where each ant is identified by its hash, and the manifest contains both a complete list of per-segment hashes and a final file hash computed over the entire reconstructed content.

This structure parallels a Merkle tree in that the file hash serves as a root authenticator, while the individual ant hashes function as leaf authenticators, enabling independent verification of each segment before reassembly. Although the manifest lacks intermediate hash nodes, it preserves the essential Merkle property: any modification to a segment produces a mismatch in its corresponding leaf hash, which consequently invalidates the reconstructed file hash, providing tamper evidence and enabling piecewise validation during parallel retrieval. ANTOR extends these foundations with selective access controls and integration with semantic metadata, producing a design optimized for hybrid knowledge-storage environments.

From a systems architecture standpoint, ANTOR establishes a resilient and adaptive data plane that operates independently of centralized tracking mechanisms. Its design incorporates advanced strategies for retry logic, peer selection, and verification, enhancing resilience against network churn and adversarial interventions. The manifest-driven model facilitates reliable reconstruction across diverse heterogeneous sources. The implementation of a fixed-size thread pool coupled with timeouts mitigates the risk of resource exhaustion and ensures stable, predictable performance.

VIII. Applications and Ecosystem Role

Within the ANNE framework, ANTOR supports essential workflows:

• Media and Document Sharing: Distribution of images, videos, and documents without central infrastructure, leveraging swarm retrieval for efficiency.
• Supplementary Storage: Handling of large, unstructured payloads that extend semantic knowledge structures, such as datasets or application assets.
• Private Replication: Secure copying among authorized participants for backup and availability, enforced by per-peer approval lists.
• Application Delivery: Propagation of interface components, datasets, and operational resources across the network, enabling decentralized software distribution.

ANTOR file transfer protocol’s close coupling with file indexing, personal annode hosting, and unstructured data distribution forms a unified storage and transfer continuum, enabling scalable, self-regulating information ecosystems.

IX. Closing Remarks

ANTOR is a purpose-built protocol for secure, peer-to-peer file transfer within the ANNE ecosystem. By combining segmented transmission (ants), manifest-based coordination (ANTOR files), and encrypted swarming with explicit access controls and hypergraph integration, it enables protected and efficient data movement across a global mesh of independent nodes. Files circulate not through remote intermediaries but via authenticated peer relationships, preserving participant autonomy while enriching the shared pool of accessible knowledge. As a component of the ANNE stack, ANTOR completes the vision of a fully decentralized infrastructure where all data, whether semantic or unstructured, is sovereign, discoverable, and verifiable.