The filecoin network consists of hundreds of storage providers distributed around the world. Content addressing and encrypted storage prove that the data is correctly and safely stored on the miner’s hardware for a long time, thus creating a powerful and reliable service.
This paper describes the two transaction types in filecoin, storage transaction and retrieval transaction, and explains their life cycle in detail. It also explains how cryptography proof is used to verify whether the participants in the system perform their duties as promised.
Data on filecoin
To store files on filecoin, users must first import files in their local filecoin node. This step produces a data CID — the content identifier, which describes the unique ID of the content. After that, the data is passed on to the miners.
Importing data into local filecoin node can be done by Lotus client Import command. Remember the CID of the generated data (which can also be obtained on the local node later), because it will be used when retrieving data from miners in the future.
After importing the data into the local node, the user needs to initiate a transaction. This is done by using the lotus client deal command. The command takes a data CID as input, generates a filecoin piece, and interactively guides the user to complete the storage transaction process.
Filecoin piece is the main negotiation unit for users to store data on filecoin network. Filecoin piece does not have a specific size, but takes the sector size as the upper limit, which is controlled by network parameters. If a filecoin piece is larger than the sector size supported by the miner, it must be divided into more fragments so that each fragment fits into a sector.
Filecoin Piece Each filecoin piece is a car file, which contains an ipld DAG with corresponding data CID and piece CID.
Car is content addressable file. Each car file is a serial representation of ipld DAG, that is, its data blocks are concatenated together with the header information (and root CID) describing the DAG diagram.
When users want to store files in the filecoin network, they first need to use unixfs to make the ipld DAG of files (this is the function of lotus client Import command). The hash representing the DAG root node is an IPFs style CID, which is called data CID.
Unixfs is a protobuf based format for describing files, directories and soft links in IPFs. In filecoin, unixfs is the file format standard, and files are submitted to the filecoin network in this format.
The generated car file is supplemented with extra zeros so that it can be written as a binary Merkle tree.
Storage transaction process
Users access data through transactions in the filecoin network. Participants in the network, including miners (suppliers) and users (demanders), interact with each other by storing and retrieving transactions.
The life cycle of storage transaction is as follows:
The user first determines the miner and its pricing, that is, the price per epoch (30 seconds) per gib that the miner wants to receive in order to accept the transaction, and the unit is attofil. Currently, the minimum period for a transaction in filecoin is 180 days.
You can query the synchronized nodes through the JSON RPC API, list all the currently active miners, and use the filecoin. Statelistminers method. You can choose according to the miner’s reputation and ability in the network. The reputation index of miners has not entered the filecoin agreement.
After selecting the miner, you can use methods such as filecoin. Stateminerinfo to obtain the miner’s peerid, which is used to establish a secure connection with the other party in the libp2p protocol.
Next, you can use the filecoin. Clientqueryask method to get a signed storageask.
The results include the details of the transaction that the miner is willing to accept, such as the range of accepted filecoin piece sizes and the price per gib per epoch. It should be noted that a storage transaction matching the miner’s storage requirements is only a prerequisite, but it is not enough to ensure that the transaction is accepted – the storage provider may run its own decision logic later.
- Price negotiation and data transmission
At this stage, both parties reach an agreement on the terms of the transaction, such as transaction cost, transaction period, transaction start time, etc.
The data is then sent from the user to the miner.
Publishing transactions on the chain through publishstoragedeals message makes the storage provider responsible for the transaction disclosure.
Once the transaction is released on the chain, it will be handed over to the mining subsystem, packaged into sectors, then packaged, and then continuously proved available.
The storage and mining subsystem ensures that the data of the filecoin network is effectively saved by the miners, and:
Participate in the filecoin storage market, undertake user data, participate in storage transactions.
Participate in the consensus of filecoin storage computing power, verify and produce blocks, make the filecoin blockchain grow, and get block rewards.
The system monitors the following processes: Commit to new storage and register new sectors
In order to register a sector in filecoin, the miner must encapsulate the sector_ Encapsulation_ The process needs a lot of calculation to produce a unique representation of data in the form of proof, that is, copy proof or porep. Once the proof is generated, the miner compresses it and submits the result to the blockchain. This proves that the miners did copy the data they agreed to store.
To prove that the storage is continuously available, all storage miners need to continuously submit certificates on the chain to verify that the sector is completely stored.
Declare a storage failure and recover from the failure. If the above certificate required by the sector is not submitted successfully, it will cause a failure and the miner will be punished.
For the consideration of storage miners and users, as mentioned above, storage transactions are activated and encapsulated only after they are published on the chain. This is important because publishing deals lock users’ funds in the chain. Only in this way, after the data is stored in the sector, the miners’ income can be guaranteed.
Publishing transactions on the chain can be regarded as signing contracts, and encapsulating and activating transactions can be regarded as starting to make commitments.
From the user’s point of view, if you want to use filecoin to store data, the transaction generally goes through the following stages:
Transaction into the gold, the user will lock the funds into escrow
Offer a deal to the miners
Check the intention to accept the transaction
The data is transferred to the miner to make the transaction, which is done through the graphsync protocol. Graphsync is a protocol to synchronize ipld graph between nodes. The protocol allows the local node to send a request to the remote node to obtain the result of searching by selector on the ipld graph of the remote node. Lotus uses IPFs / go graphsync, the implementation of graphsync protocol.
Check acceptance – make sure the miner has accepted the deal and posted it on the chain.
Encapsulation – the transaction is already on the chain and the miner is encapsulating the sector containing the transaction.
Active – the transaction has been sealed and is active. From here on, storage providers / miners should periodically prove that they continue to store data.
From the miner’s point of view, by storing the user’s data to provide services, the transaction roughly goes through the following stages:
Verify transaction – receive the transaction proposal and check its parameters (size, price, etc.).
Check for lock in Funds – make sure the user has lock in funds and can pay for the transaction.
Waiting for data – receiving transaction data from customers.
Provide collateral for transactions for chain transactions.
Publish transactions on the chain.
Encapsulating sector activation transactions, storage providers (miners) regularly submit windowpest to prove that they are continuously storing data.
Retrieval transaction is different from storage transaction, which uses payment channel and is mainly completed under the chain. Data transmission is priced by quantity, and users pay the miners gradually in the process of data transmission. In the whole process, only the creation of payment channel and exchange voucher involves the interaction with filecoin blockchain.
The overall process is as follows:
Discovery – users find out the miners who have the data they need and ask them for the details of the search quotation – price per byte, unsealing price, payment interval.
Set up a payment channel – the user needs to set up a payment channel with the miner (if it doesn’t exist yet).
Data transmission and payment – miners send data to users until they need to pay. When a certain threshold is reached, payment processing will be required, and then data transmission will continue. Depending on whether miners have data in their block store, they may need to unseal the data first – an unconventional and non instantaneous operation, which is the reverse of the encapsulation described in the storage transactions section.
At this time, the user has not obtained the complete data.
Space time proof
The above section quickly lists many details that make filecoin unique, and provides guarantee for user data in probability. This section introduces the two kinds of proofs used by filecoin and explains how they become part of the protocol and the problems they solve.
The spatiotemporal proof (PST) is the proof submitted by miners to the filecoin network, proving that it is continuing to store the only copy of data for the network.
At present, there are two types of spatiotemporal proofs in filecoin
Winningpest is a mechanism to reward storage miners for their contribution to the filecoin network. At the beginning of each epoch, a small number of storage miners are selected, and each miner digs out a new block. The specific requirement is that these miners submit the compressed storage proof of the specified sector. Each elected miner who successfully creates a block will receive a fil (block Award) and the opportunity to charge other filecoin participants who want to include information in the block.
If the storage miners fail to meet the requirements within the necessary time window, they will lose the opportunity to produce blocks, but they will not be punished for not producing blocks.
Windowpost is a mechanism for filecoin blockchain to audit the promises made by storage miners.
Every storage miner should maintain their commitment. These sectors contain transactions with users or may be empty. The latter is called committed capacity, that is, miners can make a capacity commitment to fill a sector with any data, not user data. Maintaining these sectors allows storage miners to prove that they are reserving space in the generation network.
Each day is divided into several time windows. At present, there are 48 time windows, each lasting for 30 minutes (60 epochs, because one epoch is equal to 30 seconds).
Each miner’s commitment sector is divided into several groups, and each group corresponds to a time window.
Within a time window (30 minutes), each storage miner must submit a spatiotemporal proof for each sector in that time window. This requires accessing each sector of the time window at any time, generating ZK snark proof, and publishing the join block to the filecoin blockchain. In this way, each sector of the committed storage will be audited at least once every 24 hours, and a permanent, verifiable and public record will be kept to prove the conservative commitment of each storage miner.
In the example above, you can see that a miner should submit window time and space proofs in deadline 0 (> 16TB), deadline 1 (< 8tb) and deadline 2 (< 8tb), most of which are in deadline 0. The deadline of each miner is random. For this particular miner, it starts from epoch 1635, epoch 1695 and epoch 1755 respectively. You can check these deadlines and more details about miners on the spacegap tool.
The filecoin network expects the stored data to be continuously available. Failure to commit windowcast for a sector will result in a failure, and the storage miner supplying the sector will be punished. This encourages the healthy operation of storage miners.
When the certificate is not included in the filecoin blockchain within the time limit due to network connection loss, storage failure or malicious behavior, the failure occurs.
When a sector is registered as a fault, the filecoin network will punish the storage miners who should have stored the sector; In other words, punishment assessment will be carried out for the miner’s failure to continuously store (paid from the miner’s advance collateral).
The sector failure fee is divided into three categories
Sector failure fee: it shall be paid by each sector in failure state every day. The size of the fee is slightly higher than the block reward that the sector is expected to receive every day. If a sector fails for more than 2 consecutive weeks, the sector will pay termination fee and be removed from the blockchain state.
Sector fault detection fee: This is a one-time payment fee, if the fault is detected by the chain mechanism rather than the miner’s honest report. Considering the probability of space-time proof checking, the charge is set as the block reward of the corresponding sector in several days.
Sector termination fee: a sector may terminate before the expiration date due to failure or miner’s initiative. In principle, the termination fee charged is equivalent to the current revenue generated by a sector, and does not exceed a limit, so as not to hinder the long-term sector.
This paper describes some concepts about storing and retrieving data on filecoin, protocols used by users and miners for accessing data, and various kinds of proofs and guarantees involved in these processes.
From the perspective of users and miners, the process of storage transaction and retrieval transaction is introduced in detail; As well as the punishment measures of filecoin protocol when one party has malicious behavior.
To sum up, this paper outlines how the filecoin protocol manages the filecoin network to make it a reliable and trustless decentralized storage network.