IPFS Found Guideline

This document will explain the data source, the data processing, and the steps for usersto retrieve, view, and use the data.

About Data Source

We have adopted and plan to use the following datasets as data sources: The datasets stored in the IPFS Found are from the followling source, and we will store more and more real data in the future.

Data sources:
Data processing
(1) Download the data via 'wget' command from the link provided by dataset.md e.g. wget -c
(2) Use 'split' command to split the file into small pieces of the same size (30M ~ 30G) and use the split naming convention. we will suffix the original file name, for example, we propose to split a 90G file called ' mysql-2019-06-01.tar.gz' split -b 30G mysql-2019-06-01.tar.gz mysql-2019-06-01.tar.gz Three small files will be generated, and the filename rule you can see here(https://man7.org/linux/man-pages/man1/split.1.html):

mysql-2019-06-01.tar.gza

mysql-2019-06-01.tar.gzb

mysql-2019-06-01.tar.gzc

(3) Use the API provided by the Filecoin tool - 'PowerGate'(https://github.com/textileio/powergate) to interaction with Filecoin & IPFS:
  • Import data :

    lotus client import [ command options ] [ inputPath ]

  • Initialize storage deal with a miner :

    lotus client deal < --from address> [ dataCid miner price duration ]

  • List storage market deals :

    lotus client list-deals

About User
Data Retrieval

I f you want to look for some files, you can search it by filename or dataCID.

Data Display

When you expect to view a file, you can search the file by filename or dataCID. And you can also click the specific files.

Data Use

If you want to the original file, you can download all of the data pieces to your local disk and merge them. The merge method is as follws:

e.g
  • The original file : mysql-2018-11-01.tar.gz
  • The file pieces : 
    mysql-2018-11-01.tar.gza,
    mysql-2018-11-01.tar.gzb, 
    mysql-2018-11-01.tar.gzc
  • File merge method : 
  • cat mysql-2018-11-01.tar.gz* > mysql-2018-11-01.tar.gz