elective-stereophonic
elective-stereophonic
MofNfs - fully encrypted decentralized M of N file storage
Please login or register.

Login with username, password and session length
Advanced search  

News:

Latest Stable Nxt Client: Nxt 1.11.15 | Latest Experimental Nxt Client: Nxt 1.12.0e

Pages: 1 2 [All]

Author Topic: MofNfs - fully encrypted decentralized M of N file storage  (Read 6340 times)

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
MofNfs - fully encrypted decentralized M of N file storage
« on: October 07, 2014, 08:15:28 pm »

 I decided to make two new API calls that allow for cloud storage of files. They are massively encrypted and also M of N is supported to deal with hash collisions, sybil attacks, offline nodes, etc. With the proper M and N settings, I think this will be quite a resilient file storage appropriate for the files you just cant lose. The comms with the cloud are via the DHT API from this weekend and the L parameter is for the max number of onion layers to use and all the packets are the same size, so there is no leakage based on packet size.

Now I am not sure what all the other decentralized storage projects are doing and I am sure what I did today is just a small portion of a full system. Still, after I debug it tomorrow, it will be an easy way to safely put things in the cloud.

char *savefile[] = {  "filename", "L", "M", "N", "usbdir", "password", 0 };
char *restorefile[] = { "filename", "L", "M", "N", "usbdir", "password", "destfile", "sharenrs", "txids", 0 };

./BitcoinDarkd SuperNET '{"requestType":"savefile","filename":"<file to save>","L":0,"M":1,"N":1,"usbdir":"<dir for backups>","password":"<can be 4char PIN>"}'

The savefile will print (and save in usbdir) the required sharenrs and txids JSON fields to use for the restorefile.
The "destfile" field is where the file will be reconstructed.

If the "usbdir" parameter is set, then local backups are made (highly recommended!) and it is used to check the data coming back from the cloud. After you verify that the cloud has a proper copy, then you can partition the various parts from the usbdir directory to various places to have two full backups, one under your local control and one in the cloud.

The max value for N is 254 and M has to be less than or equal to N. The M of N parameters are independent of the "password" field. If you are using M of N, then unless the attacker gets a hold of M pieces, they wont be able to reconstruct the file. Without the txid list, the attacker wont know how to reconstruct the file.

But why take any chances. so I made the password field use an iterative method to create what I think is a pretty practical encryption method, which is based on the name of the file, your pubNXT acct passphrase and the password itself. The length of the password determines the number of ciphers that are applied

        namehash = calc_txid(name,strlen(name));
        len = strlen(password);
        passwordhash = (namehash ^ calc_txid(keygen,strlen(keygen)) ^ calc_txid(password,len));
        for (i=0; i<len; i++)
        {
            expand_nxt64bits(key,passwordhash);
            cipherids = (password % NUM_CIPHERS);  // choose one of 18 ciphers
            privkeys = clonestr(key);
            if ( i < len-1 )
                passwordhash ^= (namehash ^ calc_txid(key,strlen(key)));
        }
   
Since the keygen is the pubNXT password, which in turn is a dumpprivkey for a BTCD address, this assures high entropy and the filename being encrypted is added to the passwordhash so that different files will have different encryption keys. By using the password to modify the initial password hash and to determine the number of ciphers and their sequence creates a lot of impact from even a short password, like a PIN

When M of N is combined with password, the attacker would need to get a hold of the name of the file, M fragments, the list of txids, the randomly generated sharenrs array and the password you used. Unless your computer is totally compromised and you divulge your short password, this seems like a pretty good level of security.

Now with the DHT there is the chance of collision, sybil attacks, inaccessible nodes, etc. I think using M of N side steps all of these issues. Also, the txid (calculated like NXT does) is based on the contents being stored, so it would take a lot of computation to be able to even get control of the nodes needed to block access to any specific content and near impossible to spoof anything. Maybe someone can come up with a sybil attack that can be done? However, without knowing the hash values of all the fragments, where will the sybils setup their attack? And will they be able to invalidate M copies that they dont know the txid for?

I hope for assistance in testing this API as it is quite important. Also, any method of attack that can be used against this would help me design a better system

James

###
The following are the ciphers:
    "aes","blowfish","xtea","rc5","rc6","saferp","twofish","safer_k64","safer_sk64","safer_k128",
    "safer_sk128","rc2","des3","cast5","noekeon","skipjack","khazad","anubis","rijndael"
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

bcdev

  • Hero Member
  • *****
  • Karma: +162/-22
  • Offline Offline
  • Posts: 666
    • View Profile
Re: MofNfs - fully encrypted decentralized M of N file storage
« Reply #1 on: October 07, 2014, 08:26:01 pm »

What's the difference between "aes" and "rijndael"?
« Last Edit: October 07, 2014, 08:41:07 pm by bcdev »
Logged

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: MofNfs - fully encrypted decentralized M of N file storage
« Reply #2 on: October 07, 2014, 08:39:05 pm »

What's the difference between "aes" and "rijndael"?
const struct ltc_cipher_descriptor rijndael_desc =
{
    "rijndael",
    6,
    16, 32, 16, 10,
    SETUP, ECB_ENC, ECB_DEC, ECB_TEST, ECB_DONE, ECB_KS,
    NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL
};

const struct ltc_cipher_descriptor aes_desc =
{
    "aes",
    6,
    16, 32, 16, 10,
    SETUP, ECB_ENC, ECB_DEC, ECB_TEST, ECB_DONE, ECB_KS,
    NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL
};

Apparently just the name!
since aes is the most popular, I guess this doubles the chance for using it

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

PoofKnuckle

  • Hero Member
  • *****
  • Karma: +111/-7
  • Offline Offline
  • Posts: 849
  • Your Concern is Noted.
    • View Profile
    • NXT FreeMarket
Re: MofNfs - fully encrypted decentralized M of N file storage
« Reply #3 on: October 07, 2014, 08:52:03 pm »

So are files saved redundantly across multiple hard drives belonging to other users? (Or encrypted portions of files, more likely...)

Is the source of the file private?

Logged
NXT: NXT-WBVP-83YM-FRB3-9MQLY
BM-NBTVHaA41Ejh2G6krD9p5vM62vk84isM

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: MofNfs - fully encrypted decentralized M of N file storage
« Reply #4 on: October 07, 2014, 09:04:20 pm »

So are files saved redundantly across multiple hard drives belonging to other users? (Or encrypted portions of files, more likely...)

Is the source of the file private?
actually currently it is just in RAM, but saving on HDD is the next step
if encryption is not specified, it is stored in plaintext
each file fragment is stored from a specific acct, so if you are doing this from a publicly known acct, it wont be private
but if you use a private acct, then it will be

also the file fragments are stored based on the hash of the content. and there is no other info, only the creator of the file will know. it is not practical to be making millions of random queries to find file fragments and trying to piece them together

similar problem to mining NXT accts. a lot of work and you might stumble into a fragment and then what? if it is encrypted maybe you can eventually brute force it and now you have a fragment but only the nodes that stored it would know who it came from. so the attacker would need to have a large number of nodes and be logging all this.

tl:dr I dont think it will be practical to correlate encrypted file fragments to accts if the proper precautions are taken
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

Taek

  • Jr. Member
  • **
  • Karma: +6/-1
  • Offline Offline
  • Posts: 56
    • View Profile
Re: MofNfs - fully encrypted decentralized M of N file storage
« Reply #5 on: October 07, 2014, 11:11:34 pm »

Is this incentivized at all? Or is it more like Tor where people are storing things out of charity? What's to stop someone from going offline?

Not critical questions depending on what your goals for the system are. Just trying to understand it better.
Logged

CRServers

  • Jr. Member
  • **
  • Karma: +7/-1
  • Offline Offline
  • Posts: 39
    • View Profile
    • CRServers Internet Hosting Services
Re: MofNfs - fully encrypted decentralized M of N file storage
« Reply #6 on: October 07, 2014, 11:36:09 pm »

I hope for assistance in testing this API
It will be nice to see somebody stepping to the plate, taking this challenge, and building a "NXTbox" on top  of it ;)
I'll gladly sell my Storj coin and move it into NXT

Rodrigo
Logged
CRServers
Internet Hosting Services in Costa Rica

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: MofNfs - fully encrypted decentralized M of N file storage
« Reply #7 on: October 07, 2014, 11:37:37 pm »

Is this incentivized at all? Or is it more like Tor where people are storing things out of charity? What's to stop someone from going offline?

Not critical questions depending on what your goals for the system are. Just trying to understand it better.
Nothing stops someone from going offline

The plan is to have ~100 community funded servers that can be relied on to be there. Of course, no assurance that all parts of all your files will reside in these community servers, nor that the community funding will be there forever, but for now it looks like we can fund this from the staking revenues.

I am just building the lowest level API for now. The best way to test the DHT was to build a save/restore file functionality. once the save/restore is debugged, then it will be onto the next step.

Currently, I am thinking that people would be able to use as much as they contribute, eg. if you allow 100 MB to be stored on your computer, you get to store 100 MB of your data (redundancy counts against the total).

Clearly, some people will want to consume more space than they contribute and for this there will be a realtime auction market, so people can set the price and if somebody wants to use the space, they would need to put a bid that is high enough to be accepted. Let the market decide the price is my policy.

The billing side is not solved yet, I just want to get the save/restore working first. One step at a time.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

lucky88888

  • Hero Member
  • *****
  • Karma: +42/-14
  • Offline Offline
  • Posts: 694
  • NXT-E328-UJDF-KTGH-9C6YQ
    • View Profile
Re: MofNfs - fully encrypted decentralized M of N file storage
« Reply #8 on: October 08, 2014, 01:34:58 am »

Quote
Nothing stops someone from going offline

The plan is to have ~100 community funded servers that can be relied on to be there. Of course, no assurance that all parts of all your files will reside in these community servers, nor that the community funding will be there forever, but for now it looks like we can fund this from the staking revenues.

then files uploaded to this network can't be guaranteed?
people throw old computer away and hdd fails, the unlucky uploader's data inside those will not be recoverable?
Logged
NXT-E328-UJDF-KTGH-9C6YQ
8897013707391239174

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: MofNfs - fully encrypted decentralized M of N file storage
« Reply #9 on: October 08, 2014, 01:42:08 am »

Quote
Nothing stops someone from going offline

The plan is to have ~100 community funded servers that can be relied on to be there. Of course, no assurance that all parts of all your files will reside in these community servers, nor that the community funding will be there forever, but for now it looks like we can fund this from the staking revenues.

then files uploaded to this network can't be guaranteed?
people throw old computer away and hdd fails, the unlucky uploader's data inside those will not be recoverable?
if they use M of N at 1 of 1, then there will "only" be K copies of the data, with K probably around 10 to 20. So if all K of these nodes at the same time go offline, and there was only one copy, then yes it is gone.

it is a matter of statistics.
If you use M of N at 1 of 254, then there will be 254*K copies, so if all 254*K nodes go offline, then the data is lost. What are the odds of 2540 nodes all going offline the same day?

also, if there are a dozen coordinated EMP strikes at key locations around the world, then probably most of the data will be wiped out.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

lucky88888

  • Hero Member
  • *****
  • Karma: +42/-14
  • Offline Offline
  • Posts: 694
  • NXT-E328-UJDF-KTGH-9C6YQ
    • View Profile
Re: MofNfs - fully encrypted decentralized M of N file storage
« Reply #10 on: October 08, 2014, 02:10:10 am »

okay that's pretty cool. i had no idea of what MofNs are.
files are secure enough but this would be the most efficient way? if storing 20copies of everything, isn't that alot of wasted storage?

say we have 20peoples online and those 20people each offer 1mb of storage. then only 1 is able to store 1mb and all those 19 people will have no storage even if they all suppose to get 1mb from the 1mb they are offering. we could say cloud storage on fractional reserve.

im sorry if i sounded ridiculous, im just curious how the tech works.  ;D
Logged
NXT-E328-UJDF-KTGH-9C6YQ
8897013707391239174

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: MofNfs - fully encrypted decentralized M of N file storage
« Reply #11 on: October 08, 2014, 02:19:32 am »

okay that's pretty cool. i had no idea of what MofNs are.
files are secure enough but this would be the most efficient way? if storing 20copies of everything, isn't that alot of wasted storage?

say we have 20peoples online and those 20people each offer 1mb of storage. then only 1 is able to store 1mb and all those 19 people will have no storage even if they all suppose to get 1mb from the 1mb they are offering. we could say cloud storage on fractional reserve.

im sorry if i sounded ridiculous, im just curious how the tech works.  ;D
bittorrent works this way, not sure the number of copies, popular items get cached to quite a few nodes

With K of 20, then for every MB of storage you contribute you would get 50KB
so this is not for storing big files, but rather super important stuff you never want to lose

Maybe K of 7 is enough, but if K becomes too small, it becomes more likely than the EMP scenario :)
basically cloud storage is more reliable due to its redundancy and more secure since nobody actually knows where the data is at any given time. just the process of searching for it ends up replicating it!

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

lucky88888

  • Hero Member
  • *****
  • Karma: +42/-14
  • Offline Offline
  • Posts: 694
  • NXT-E328-UJDF-KTGH-9C6YQ
    • View Profile
Re: MofNfs - fully encrypted decentralized M of N file storage
« Reply #12 on: October 08, 2014, 02:30:16 am »

Quote
With K of 20, then for every MB of storage you contribute you would get 50KB
so this is not for storing big files, but rather super important stuff you never want to lose

do the storage of files gets shuffled around the network to prevent failures across all 20copies? if the storage are static, with time, no matter how many copies you put up, one by one will be lost. or do they just duplicate another copy every time a node that stores it fails just to make sure always 20copies are online?
Logged
NXT-E328-UJDF-KTGH-9C6YQ
8897013707391239174

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: MofNfs - fully encrypted decentralized M of N file storage
« Reply #13 on: October 08, 2014, 02:33:22 am »

Quote
With K of 20, then for every MB of storage you contribute you would get 50KB
so this is not for storing big files, but rather super important stuff you never want to lose

do the storage of files gets shuffled around the network to prevent failures across all 20copies? if the storage are static, with time, no matter how many copies you put up, one by one will be lost. or do they just duplicate another copy every time a node that stores it fails just to make sure always 20copies are online?
daily refresh keeps a minimum of K copies around every day
so if a node that used to have a copy disappears, the next day some other node would be closest to the hash value
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

costa2439

  • Full Member
  • ***
  • Karma: +5/-0
  • Offline Offline
  • Posts: 109
    • View Profile
Re: MofNfs - fully encrypted decentralized M of N file storage
« Reply #14 on: October 08, 2014, 10:24:36 am »

What's your position on sianote?

considering jl777hodl has 400 pcs
Logged

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: MofNfs - fully encrypted decentralized M of N file storage
« Reply #15 on: October 08, 2014, 10:49:16 am »

https://forum.thesupernet.org/index.php?topic=154.msg1179#msg1179

got savefile and restorefile debugged, at least locally, so if the DHT calls work then it should work in the cloud

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: MofNfs - fully encrypted decentralized M of N file storage
« Reply #16 on: October 08, 2014, 10:51:12 am »

What's your position on sianote?

considering jl777hodl has 400 pcs
I dont know about sianote details, you should ask Taek. they still have a money left, so if they determine they cant do it, maybe they will do a buyback?

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

costa2439

  • Full Member
  • ***
  • Karma: +5/-0
  • Offline Offline
  • Posts: 109
    • View Profile
Re: MofNfs - fully encrypted decentralized M of N file storage
« Reply #17 on: October 08, 2014, 11:05:34 am »

What's your position on sianote?

considering jl777hodl has 400 pcs
I dont know about sianote details, you should ask Taek. they still have a money left, so if they determine they cant do it, maybe they will do a buyback?

James

sorry, I don't understand
Logged

costa2439

  • Full Member
  • ***
  • Karma: +5/-0
  • Offline Offline
  • Posts: 109
    • View Profile
Re: MofNfs - fully encrypted decentralized M of N file storage
« Reply #18 on: October 08, 2014, 11:07:42 am »

I mean, if a conflict between these two projects, the investment should have been sold to avoid devaluing the background jl777hdl
Logged

coinomat

  • Hero Member
  • *****
  • Karma: +214/-18
  • Offline Offline
  • Posts: 1520
    • View Profile
Re: MofNfs - fully encrypted decentralized M of N file storage
« Reply #19 on: October 08, 2014, 11:13:28 am »

Sorry if I'm missing something, but the nodes will be funded from centralized source? is there a way to make people host a node and earn from doing it?
Some kind of proof of bandwidth. I've been thinking for some time about such a system, couldn't find a stable way to do it.
Logged
Time to go further

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: MofNfs - fully encrypted decentralized M of N file storage
« Reply #20 on: October 08, 2014, 11:19:46 am »

I mean, if a conflict between these two projects, the investment should have been sold to avoid devaluing the background jl777hdl
you are not making sense
when did I ever say that JLH would not have investments in assets that do things in the same area as other tech I do?
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: MofNfs - fully encrypted decentralized M of N file storage
« Reply #21 on: October 08, 2014, 11:23:19 am »

Sorry if I'm missing something, but the nodes will be funded from centralized source? is there a way to make people host a node and earn from doing it?
Some kind of proof of bandwidth. I've been thinking for some time about such a system, couldn't find a stable way to do it.
just 50 to 100 nodes will be community funded
I plan to allow people to use as much of the cloud storage as they contribute
excess consumers would have to pay, net suppliers would share all such payments proportional to their contribution
probably an InstantDEX auction process for people that want to buy more storage

the issue is how to release more inventory of storage in sync with payments, but first I want to get the fundamental tech solid. I did only start this last Friday, so monetization has not really been a priority

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: MofNfs - fully encrypted decentralized M of N file storage
« Reply #22 on: October 11, 2014, 08:12:55 am »

I figured out how to do multisig, or at least multisigning using curve25519.
https://forum.thesupernet.org/index.php?topic=154.msg1262#msg1262

A, B, C are the privatekeys
a, b, c are the publickeys

completed on computer A:
sha256_key(xor_keys(seed,curve25519(A,curve25519(B,c))))
sha256_key(xor_keys(seed,curve25519(A,curve25519(C,b))))

completed on computer B:
sha256_key(xor_keys(seed,curve25519(B,curve25519(A,c))))
sha256_key(xor_keys(seed,curve25519(B,curve25519(C,a))))

completed on computer C:
sha256_key(xor_keys(seed,curve25519(C,curve25519(A,b))))
sha256_key(xor_keys(seed,curve25519(C,curve25519(B,a))))

All three pairs map to the same point so the outputs match and on each computer, for 2 of 3 only one of the other two computers has to respond and for 3 of 3, both of them do.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

KyLin

  • Full Member
  • ***
  • Karma: +4/-0
  • Offline Offline
  • Posts: 101
    • View Profile
    • nxtfans
Re: MofNfs - fully encrypted decentralized M of N file storage
« Reply #23 on: October 14, 2014, 06:24:11 am »

Waht about freemarket
Logged

coinomat

  • Hero Member
  • *****
  • Karma: +214/-18
  • Offline Offline
  • Posts: 1520
    • View Profile
Re: MofNfs - fully encrypted decentralized M of N file storage
« Reply #24 on: October 15, 2014, 10:59:06 am »

I thought about the following - why not use the torrent tracker technique for tracking download and upload stats and pay the participating node for their bandwidth? Users could also prepay the cloud storage service and the "tracker" would distribute the fees between the nodes based on the stats.
It would introduce centralization but only for stats and payments, and could attract users sharing their storage/bandwidths to make  money.
Logged
Time to go further

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: MofNfs - fully encrypted decentralized M of N file storage
« Reply #25 on: October 15, 2014, 11:05:51 am »

I thought about the following - why not use the torrent tracker technique for tracking download and upload stats and pay the participating node for their bandwidth? Users could also prepay the cloud storage service and the "tracker" would distribute the fees between the nodes based on the stats.
It would introduce centralization but only for stats and payments, and could attract users sharing their storage/bandwidths to make  money.
I hope eventually to be able to decentralize either the buying or selling side, or both
but one step at a time
at first I am thinking of a randomized audit process
the nodes are queried as to what they are storing and this is tallied for the billing

« Last Edit: October 15, 2014, 11:08:38 am by jl777 »
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

allwelder

  • Hero Member
  • *****
  • Karma: +196/-13
  • Offline Offline
  • Posts: 1867
  • NxtChina.org
    • View Profile
    • NxtChina.org
Re: MofNfs - fully encrypted decentralized M of N file storage
« Reply #26 on: October 16, 2014, 01:16:28 am »

Waht about freemarket
This is Unity thread.
Freemarket is here.
Logged
NxtChina |Weibo |Twitter Donation welcomed:NXT-APL9-66GU-K8LY-B3JJJ
Pages: 1 2 [All]
 

elective-stereophonic
elective-stereophonic
assembly
assembly