elective-stereophonic
elective-stereophonic
Sia Official Nxt Thread singapore
Please login or register.

Login with username, password and session length
Advanced search  

News:

Latest Stable Nxt Client: Nxt 1.11.15 | Latest Experimental Nxt Client: Nxt 1.12.0e

Pages: 1 2 [3] 4 5 ... 8  All

Author Topic: Sia Official Nxt Thread  (Read 42522 times)

gs02xzz

  • Hero Member
  • *****
  • Karma: +56/-12
  • Offline Offline
  • Posts: 1101
    • View Profile
Re: Sia Official Nxt Thread
« Reply #40 on: June 23, 2014, 01:36:54 pm »

Update, I've devised a new way to do storage proofs, but I'm not 100% certain that it's cryptographically secure. It looks like it should work, but I'm nervous that I've missed something simple, an easy way to cheat the system.

How the new file proofs work:

File segments are split into 4kb pieces, exactly N 4kb pieces.
A merkle tree is made of these 4kb pieces, only the final hash is kept. As an example, I'll use a 4 piece segment, 16kb total.

The bottom level is 4 4kb pieces, w.x.y.z
The next level up is their hashes, a.b.c.d
The next level up is each pair together, A=h(a+b), B=h(c+d), for the level A.B
The top level, O, is the hash of A and B, O=h(A+B)

The network randomly selects a piece for you to prove that you have, knowing only O. The network wants w
So, to prove that you have w, you provide w,b, and B. The network then hashes w to get a, then hashes a+b to get A, then hashes A and B to get O.

This same sequence can be performed with any other of w.x.y.z, except that the order in which the hashing is done changes depends on which piece you are proving for.

The problem is that I've been having trouble proving that it's secure. I thought that the proof already existed somewhere, Merkle Trees are pretty well established. The tricky part is that the network only knows about O, and the person attempting the proof is allowed to provide every other component. This should be a normal feature of a Merkle Tree but again I'm having trouble finding/devising a proof of security. I was hoping that someone here could help me out with a link or something.

Thanks!

Bump. Can anyone help to locate a link for this?
Logged
Nxt Mission is to commercialize the crypto technology and build new commerce and society.

Taek

  • Jr. Member
  • **
  • Karma: +6/-1
  • Offline Offline
  • Posts: 56
    • View Profile
Re: Sia Official Nxt Thread
« Reply #41 on: June 26, 2014, 05:43:11 am »

Status Report

As far as new implementation goes, there are only 3 things that remain before the alpha:

+ proof of storage is only 75% complete
+ the client isn't ready yet either
+ support for downloading files (lol nobody needs downloads... right? it's the uploading that counts)

Before the launch though:

+ need to update the whitepaper
+ need to write documentation
+ things are buggy and partially untested. We're planning on releasing something that's buggy, but we can at least do a little more cleanup
+ need to make a nice launch page

Limitations of the alpha:

- Though there's a great scripting system, we've written very few scripts, and Luke is pretty much the only one who understands the bytecode powering the scripting system. Also the expense structure (which prevents infinite loops and abuse) can barely be considered capable.
- Uploading files takes 2 complete blocks before file uploads can even begin. This will change.
- There's no pricing algorithm for storage
- The client is very basic, and has no support for self made scripts. If you want to make your own scripts and test them out, you'll need to modify the source code of the client
- it's largely untested. All the major features work if you don't try to abuse them. We're not sure how well they'll tolerate abuse though. This is less a problem with the protocol and more a problem with the incompleteness of the alpha build.

But at least you'll be able to host a quorum, join a quorum, create wallets, send transactions, upload+download files, and if you don't mind messing with the source code you can also play around with the scripting bytecode. We'll tell you what files to edit.
Logged

Taek

  • Jr. Member
  • **
  • Karma: +6/-1
  • Offline Offline
  • Posts: 56
    • View Profile
Re: Sia Official Nxt Thread
« Reply #42 on: July 02, 2014, 03:59:17 am »

The first alpha release is complete. This isn't intended to be used for actual file storage, just to demonstrate what Sia might look like. Feedback is appreciated, but we are aware of a large number of bugs and points of insecurity. There's a lot that hasn't been implemented yet. Think of it more as a proof-of-concept than a traditional alpha.

Check it out at siacoin.com/download.html

There you will find an explanation on how to download and install Sia. Unfortunately, we don't have Windows binaries yet.

We have an IRC channel now, #sia-talk on freenode. We have registered a few other channels for future use, but until there's a need for them we plan on keeping them mostly dormant.

Things you can do with the current alpha:

Create wallets
Send money between wallets
Upload files to wallets (max 256 atoms, explained later)
Download files

The block speed for the alpha release is 256 seconds per block. Unfortunately, many actions such as upload take several blocks to complete, though never more than 4. This can make the production Sia a little slow to play around with. (If it really bothers you, you can recompile with a shorter interval.) The full version of Sia will have block speeds somewhere between 1 block every 2-16 hours. However, it will also support accelerated actions, which can only occur under certain but easily achieved constraints. Most people will not realize that Sia moves at a pace of 1 step every few hours, because they will be using cryptographic proofs behind the scenes to accelerate their actions. These accelerations won't be supported in Sia until the beta.

Notes:
- Empty wallets do not get deleted.
- Participants do not get kicked for not having files.
- Participants do get kicked for missing a single heartbeat. This can cause problems for unstable connections
- Sometimes participants don't join the quorum in-sync. This is rare, and the fix is to just try again.
- After creating a new quorum, you must wait a full block before adding any siblings.
- You may encounter some hard crashes. This is because a good error-handling framework is still in the works.

What are Atoms?
An atom is a unit of storage on the Sia network. Uploaders are allowed to pick their own redundancy, which means that the amount of data which can be stored in an atom is variable. At full redundancy (every sibling has the complete file), only 4kb can be stored per atom, the max file size is 1MB, and the redundancy is equal to the quorum size. (For the alpha, the quorum size is 32). At a lower redundancy (like double redundancy), an atom is 4KB * 16 hosts, which means the max file size is slightly more flexible.

Picking a Redundancy
Redundancy works by picking a value, k, which represents the number of siblings that need to remain online in order for the file to be recoverable. The amount of redundancy is QuorumSize/K. At k=32 (the current quorum size), the redundancy is 1, meaning there is exactly 1 copy of the file on the network. At k=16, there are 2 complete copies of the file on the network, and the original file can be rebuilt from any 16 siblings.

Setting up a Quorum
Right now, all quorums are pseudo-decentralized, which means that anybody can establish a quorum, and anybody can join a quorum that has extra space for siblings. To establish a quorum, run the server executable and type 'e'. This creates a new quorum bootstrapped to your current address and chosen port number. To join a quorum, type 'j', and input the address of any host in the quorum you wish to join.

Messing Around
Feel free to dig into the source code a bit. To reduce the quorum size (which reduces the block speed), change QuorumSize in src/quorum/quorum.go. To change the step length (currently set to 8 seconds, reducing this will also reduce the block speed), change StepDuration in src/participant/listen.go.

We realize right now that setting up a quorum is a bit of a pain, and that there are lots of frustrating things that can go wrong while keeping one running. This is merely a result of the early stage of the software. Later versions will be easier to use, less dependent on a stable internet connection, and will not have occasional synchronization hiccups.


Moving Forward:
The first thing we're doing is taking a 3-4 day break from coding. It's easy to get your head buried and miss out on what's happening in the rest of the world. We'll be using this time to take a broader perspective on Sia and figure out what the next best steps are.

We're also setting a deadline for a alpha v2 release in 3-4 weeks. This alpha will have:

+ increased DOS (denial-of-service) protection
+ automatic port forwarding
+ smooth, cryptographic synchronization among hosts
+ wallets that get deleted upon running out of money
+ nonces for transactions, meaning a signed transaction can only be used once
+ proof-of-storage with cheating protection and auto-cheating implemented. (Auto cheating is something you do if you don't have the file, for example if your disk corrupts.

After that we'll be looking toward building the meta-quorums, cryptographic DOS protection (8)), an instant-transaction network, and a more comfortable scripting environment.

If you have questions, you can post here, chat with us in the IRC channel, or shoot us an email.
Logged

costa2439

  • Full Member
  • ***
  • Karma: +5/-0
  • Offline Offline
  • Posts: 109
    • View Profile
Re: Sia Official Nxt Thread
« Reply #43 on: July 02, 2014, 08:34:14 am »

The first alpha release is complete. This isn't intended to be used for actual file storage, just to demonstrate what Sia might look like. Feedback is appreciated, but we are aware of a large number of bugs and points of insecurity. There's a lot that hasn't been implemented yet. Think of it more as a proof-of-concept than a traditional alpha.

Check it out at siacoin.com/download.html

There you will find an explanation on how to download and install Sia. Unfortunately, we don't have Windows binaries yet.

We have an IRC channel now, #sia-talk on freenode. We have registered a few other channels for future use, but until there's a need for them we plan on keeping them mostly dormant.

Things you can do with the current alpha:

Create wallets
Send money between wallets
Upload files to wallets (max 256 atoms, explained later)
Download files

The block speed for the alpha release is 256 seconds per block. Unfortunately, many actions such as upload take several blocks to complete, though never more than 4. This can make the production Sia a little slow to play around with. (If it really bothers you, you can recompile with a shorter interval.) The full version of Sia will have block speeds somewhere between 1 block every 2-16 hours. However, it will also support accelerated actions, which can only occur under certain but easily achieved constraints. Most people will not realize that Sia moves at a pace of 1 step every few hours, because they will be using cryptographic proofs behind the scenes to accelerate their actions. These accelerations won't be supported in Sia until the beta.

Notes:
- Empty wallets do not get deleted.
- Participants do not get kicked for not having files.
- Participants do get kicked for missing a single heartbeat. This can cause problems for unstable connections
- Sometimes participants don't join the quorum in-sync. This is rare, and the fix is to just try again.
- After creating a new quorum, you must wait a full block before adding any siblings.
- You may encounter some hard crashes. This is because a good error-handling framework is still in the works.

What are Atoms?
An atom is a unit of storage on the Sia network. Uploaders are allowed to pick their own redundancy, which means that the amount of data which can be stored in an atom is variable. At full redundancy (every sibling has the complete file), only 4kb can be stored per atom, the max file size is 1MB, and the redundancy is equal to the quorum size. (For the alpha, the quorum size is 32). At a lower redundancy (like double redundancy), an atom is 4KB * 16 hosts, which means the max file size is slightly more flexible.

Picking a Redundancy
Redundancy works by picking a value, k, which represents the number of siblings that need to remain online in order for the file to be recoverable. The amount of redundancy is QuorumSize/K. At k=32 (the current quorum size), the redundancy is 1, meaning there is exactly 1 copy of the file on the network. At k=16, there are 2 complete copies of the file on the network, and the original file can be rebuilt from any 16 siblings.

Setting up a Quorum
Right now, all quorums are pseudo-decentralized, which means that anybody can establish a quorum, and anybody can join a quorum that has extra space for siblings. To establish a quorum, run the server executable and type 'e'. This creates a new quorum bootstrapped to your current address and chosen port number. To join a quorum, type 'j', and input the address of any host in the quorum you wish to join.

Messing Around
Feel free to dig into the source code a bit. To reduce the quorum size (which reduces the block speed), change QuorumSize in src/quorum/quorum.go. To change the step length (currently set to 8 seconds, reducing this will also reduce the block speed), change StepDuration in src/participant/listen.go.

We realize right now that setting up a quorum is a bit of a pain, and that there are lots of frustrating things that can go wrong while keeping one running. This is merely a result of the early stage of the software. Later versions will be easier to use, less dependent on a stable internet connection, and will not have occasional synchronization hiccups.


Moving Forward:
The first thing we're doing is taking a 3-4 day break from coding. It's easy to get your head buried and miss out on what's happening in the rest of the world. We'll be using this time to take a broader perspective on Sia and figure out what the next best steps are.

We're also setting a deadline for a alpha v2 release in 3-4 weeks. This alpha will have:

+ increased DOS (denial-of-service) protection
+ automatic port forwarding
+ smooth, cryptographic synchronization among hosts
+ wallets that get deleted upon running out of money
+ nonces for transactions, meaning a signed transaction can only be used once
+ proof-of-storage with cheating protection and auto-cheating implemented. (Auto cheating is something you do if you don't have the file, for example if your disk corrupts.

After that we'll be looking toward building the meta-quorums, cryptographic DOS protection (8)), an instant-transaction network, and a more comfortable scripting environment.

If you have questions, you can post here, chat with us in the IRC channel, or shoot us an email.

Thank you
I have a question, does the system assigns benefits siastock decentralized?
or is managed for you
Logged

MingZhou

  • Newbie
  • *
  • Karma: +0/-0
  • Offline Offline
  • Posts: 7
    • View Profile
Re: Sia Official Nxt Thread
« Reply #44 on: July 02, 2014, 01:35:50 pm »

I have a question, does the system assigns benefits siastock decentralized?
or is managed for you

I believe it will be decentralized. The dev answered a similar question on the BTT thread before  - bitcointalk.org/index.php?topic=591283.new#new
Logged

Taek

  • Jr. Member
  • **
  • Karma: +6/-1
  • Offline Offline
  • Posts: 56
    • View Profile
Re: Sia Official Nxt Thread
« Reply #45 on: July 02, 2014, 01:55:29 pm »

I'm not exactly sure what you mean by that question, but siastock will be built into the protocol in the same way that siacoins are built into the protocol, it'll be managed automatically by the rules of the decentralized protocol.

Initial distribution though will be managed by us, which is one of the reasons there are only 1000 unsplittable sianotes in circulation - we'll be handing them out manually, and we can't be distributing to 15,000 accounts that each own $2 of sianotes.
Logged

MJ79

  • Jr. Member
  • **
  • Karma: +5/-0
  • Offline Offline
  • Posts: 92
    • View Profile
Re: Sia Official Nxt Thread
« Reply #46 on: July 08, 2014, 04:19:09 pm »

So, we've been holding off on this announcement, but I think we're at a place where we can tell everyone.

We've just signed a $375,000 agreement for 7.5% of the company equity. There's still some paperwork to sign, but we should have the cash by the end of July. Which means we're also about to be hiring one or two full time employees. We need someone who will build the frontend, and will preferably work and live in Boston. We might be able to pick up an MIT graduate. I've got a really good feeling as a whole about our investor, he's had several hit investments and is really interested in seeing Sia succeed.

Another thing that's in progress is that we're going to be involved with the MIT Bitcoin project. We haven't met anyone in person yet but we should be meeting with the guys leading the initiative in the next few weeks.

So, on the business front things are looking good.

Hi David,

So whoever this investor is actually bought corporate equity where SiaStock is NOT corporate equity.

The real question that pops  into my mind after reading this news is this:

Since I know you are planning bigger things than just decentralized cloud storage for Nebulous; Will SiaStock holders benefit from future ventures of Nebulous? In other words, will Nebulous use SiaCoin as a base for all future operations?

I think this is fairly important for short term SiaStock (SiaNote) valuation.

Thanks.
« Last Edit: July 08, 2014, 04:23:13 pm by MJ79 »
Logged

Taek

  • Jr. Member
  • **
  • Karma: +6/-1
  • Offline Offline
  • Posts: 56
    • View Profile
Re: Sia Official Nxt Thread
« Reply #47 on: July 09, 2014, 03:21:12 am »

Hi David,

So whoever this investor is actually bought corporate equity where SiaStock is NOT corporate equity.

Correct. Our investor bought corporate equity. Sianotes will not convert to corporate equity, and nobody has a claim to our corporate equity except for myself, my cofounder, and now our investor.

On a second note, we're renaming Siastock. For now, I would like to just call them Sianotes (until a more appropriate name is determined). The name siastock causes too much confusion as it suggests that siastock is corporate equity. I hope fo have a more appropriate name by the end of the week.

The real question that pops  into my mind after reading this news is this:

Since I know you are planning bigger things than just decentralized cloud storage for Nebulous; Will SiaStock holders benefit from future ventures of Nebulous? In other words, will Nebulous use SiaCoin as a base for all future operations?

I think this is fairly important for short term SiaStock (SiaNote) valuation.

Thanks.

This is an interesting question. The easy answer is no, sianote holders will not benefit from Nebulous products other than Sia. Sianotes convert to a claim of a small portion of host income, nothing more.
The more difficult question is whether future Nebulous products will use Siacoins, which may affect the value of Sianotes. We don't know, although I know that some of our current in-process ideas will probably not have their own currency, and therefore be reliant on other currencies. Siacoin may be one of these, depending on how simple integration with Sia is. (It should be quite simple, and it should be easy especially for us since we understand Sia backwards and forwards). I refuse to make any promises however. Sianotes have a very specific, single purpose that they convert to, and no guarantees of extra features.

I don't want to say too much more. We have a bunch of potential features in the works that would make Siacoin a very desirable currency, but I'm not sure how any of them will pan out. Right now we're focused entirely on getting the alpha to a user-friendly state, and getting to a place where people can actually store files on the network with a moderate expectation of security and availability. In the other thread I used this math to value the sianote:

Quote from: Taek
Currently Sianotes are being sold for ~$250 USD each. 1250 are then worth about $312,500, meaning at current network prices, the total host income for storage (which will also be about the amount of money spent on storage) is estimated to be about $62,000,000. This is over the entire history of Sia. Sianotes are far from worthless, especially if you expect Sia to sell more than $62,000,000 of storage over its entire lifetime. We (the devs) certainly do.

The biggest component that's missing from the above math is the chance of failure. If you think there's a 1% chance of Sia selling $1 billion in storage, and a 99% chance of Sia never doing more than a few thousand dollars of business, then your valuation would be $1b * 1% = $10,000,000. I honestly think that given our current position, $62,000,000 in total storage sales is under valued. I would put us closer to $100 or $200 million but there are some obvious biases there. Other people may think that other variables are important and use different metrics to value the Sianote. For example, if you think that overall Sia will flop, but in 6 months the Sianote will be three times it's current value, then you may buy sianotes now even though you think Sia as a whole will never do well.

I do that with Bitcoin. I honestly think that in 5 years Bitcoin is going to be worth less than it is today, and in 10 years it's going to be pretty much a collectors item, worth only 50 to 100 dollars each (if that), because Bitcoin is going to be replaced by something that's far superior. Maybe Sia will play a role in that, Ethereum might play a role in that, Nxt might play a role in that. Maybe I'm wrong. But regardless I have a large number of Bitcoins right now because I think by 2015 they are going to be worth much more than they are worth now. And that's good enough for me, I just have to get out before Bitcoin is properly replaced.

Happy to answer any more questions. I want to help everyone make the most informed investment decisions possible regarding Sia.
Logged

Berry

  • Jr. Member
  • **
  • Karma: +12/-26
  • Offline Offline
  • Posts: 88
    • View Profile
Re: Sia Official Nxt Thread
« Reply #48 on: July 09, 2014, 11:11:36 am »

I still don´t get it.

First of all, you have Nebulous. The shares own the company, and all earnings/winnings.

Second we have Sianotes. 87,5% of them are owned by Nebulous, right?

Third we have Siacoins. The coins are used for all payments, etc etc.  So if a lot of people are using Sia, the price of Soincoin increases.


But what about Sianotes. The only use is to distribute the 3,9% of fees?  Who gets the other 96,1%?

Logged

Taek

  • Jr. Member
  • **
  • Karma: +6/-1
  • Offline Offline
  • Posts: 56
    • View Profile
Re: Sia Official Nxt Thread
« Reply #49 on: July 09, 2014, 01:12:48 pm »

I still don´t get it.

First of all, you have Nebulous. The shares own the company, and all earnings/winnings.

Second we have Sianotes. 87,5% of them are owned by Nebulous, right?

Third we have Siacoins. The coins are used for all payments, etc etc.  So if a lot of people are using Sia, the price of Soincoin increases.


But what about Sianotes. The only use is to distribute the 3,9% of fees?  Who gets the other 96,1%?

It's 3.9% of host income. So 3.9% goes to Sianote holders, and 96.1% goes to the hosts offering storage to the network. Does that make sense?

The other thing is that 3.9% is not promised to be the final number. 1 sianote will definitely convert to 0.00039% of host income from renting storage, but Nebulous may decrease or increase the total fee depending on what we think is most intelligent from a business perspective. The fee will be set so that we can maximize the health of Sia. A higher fee means we can fund more developers and have a larger corporate engine behind Sia. A higher fee also means hosts are less likely to be content paying the fee, so we'll pick the number that makes the most sense. It's tentatively set to 3.9% but we want to make sure we have the flexibility to change that number.

But regardless of what the actual number ends up being, 1 sianote still converts to 0.00039% of host income from renting storage.
Logged

MJ79

  • Jr. Member
  • **
  • Karma: +5/-0
  • Offline Offline
  • Posts: 92
    • View Profile
Re: Sia Official Nxt Thread
« Reply #50 on: July 09, 2014, 02:39:34 pm »

This is an interesting question. The easy answer is no, sianote holders will not benefit from Nebulous products other than Sia. Sianotes convert to a claim of a small portion of host income, nothing more.
The more difficult question is whether future Nebulous products will use Siacoins, which may affect the value of Sianotes. We don't know, although I know that some of our current in-process ideas will probably not have their own currency, and therefore be reliant on other currencies. Siacoin may be one of these, depending on how simple integration with Sia is. (It should be quite simple, and it should be easy especially for us since we understand Sia backwards and forwards). I refuse to make any promises however. Sianotes have a very specific, single purpose that they convert to, and no guarantees of extra features.

I don't want to say too much more. We have a bunch of potential features in the works that would make Siacoin a very desirable currency, but I'm not sure how any of them will pan out. Right now we're focused entirely on getting the alpha to a user-friendly state, and getting to a place where people can actually store files on the network with a moderate expectation of security and availability.

Clear answer, thanks :)

And congratulations on finding your first real investor. I really do wish you guys all the luck in the world with Sia and Nebulous as a whole.  :)
Logged

Berry

  • Jr. Member
  • **
  • Karma: +12/-26
  • Offline Offline
  • Posts: 88
    • View Profile
Re: Sia Official Nxt Thread
« Reply #51 on: July 09, 2014, 04:51:56 pm »

I still don´t get it.

First of all, you have Nebulous. The shares own the company, and all earnings/winnings.

Second we have Sianotes. 87,5% of them are owned by Nebulous, right?

Third we have Siacoins. The coins are used for all payments, etc etc.  So if a lot of people are using Sia, the price of Soincoin increases.


But what about Sianotes. The only use is to distribute the 3,9% of fees?  Who gets the other 96,1%?

It's 3.9% of host income. So 3.9% goes to Sianote holders, and 96.1% goes to the hosts offering storage to the network. Does that make sense?

The other thing is that 3.9% is not promised to be the final number. 1 sianote will definitely convert to 0.00039% of host income from renting storage, but Nebulous may decrease or increase the total fee depending on what we think is most intelligent from a business perspective. The fee will be set so that we can maximize the health of Sia. A higher fee means we can fund more developers and have a larger corporate engine behind Sia. A higher fee also means hosts are less likely to be content paying the fee, so we'll pick the number that makes the most sense. It's tentatively set to 3.9% but we want to make sure we have the flexibility to change that number.

But regardless of what the actual number ends up being, 1 sianote still converts to 0.00039% of host income from renting storage.


Alright understood.

So if you really reach your goal of 62.000.000 every sianote will get:

62.000.000 * 0.00039% = 241,8 $

That´s a very high goal, and the return over the years to reach this goal is very low.
The only reason for a big rise of the value of sianotes is that you reach much more than the 62.000.000

I don´t see a big potential in Sianotes. The Siacoins can increase big, the Nebulous company could become big, but the Sianotes are limited in my opinion.

Don´t get me wrong I already bought sianotes, but thought wrong about it.

Or didn´t I get it?
Logged

Taek

  • Jr. Member
  • **
  • Karma: +6/-1
  • Offline Offline
  • Posts: 56
    • View Profile
Re: Sia Official Nxt Thread
« Reply #52 on: July 09, 2014, 06:45:50 pm »

The $62,000,000 isn't a goal. It's what would be required to justify the current price of the Sianote. If the total amount of money spent on storage in the entire history of Sia reached $62,000,000, then every Sianote would get a return of ~$250. ($241.8, I guess - didn't check the exact math). I think that it's possible that the majority of storage will be used over Sia, something that would represent billions of dollars annually. If that happened, each Sianote would get a lifetime income in the tens of thousands of dollars. That's our real goal.
Logged

Berry

  • Jr. Member
  • **
  • Karma: +12/-26
  • Offline Offline
  • Posts: 88
    • View Profile
Re: Sia Official Nxt Thread
« Reply #53 on: July 09, 2014, 08:33:40 pm »

The $62,000,000 isn't a goal. It's what would be required to justify the current price of the Sianote. If the total amount of money spent on storage in the entire history of Sia reached $62,000,000, then every Sianote would get a return of ~$250. ($241.8, I guess - didn't check the exact math). I think that it's possible that the majority of storage will be used over Sia, something that would represent billions of dollars annually. If that happened, each Sianote would get a lifetime income in the tens of thousands of dollars. That's our real goal.

Really?

For a return of 10000$ per Sianote and just one time every year --> Users have to pay 62.000.000.000 $ for storage every year.
62 Billion every year !!!
So I didn´t do any research of the global data cloud storage. But even as monoplist it is hard to believe that sia could earn that much in fees.

I don´t believe it, but don´t matter. I hope you can proof me wrong. Let´s go  ;)


Logged

Taek

  • Jr. Member
  • **
  • Karma: +6/-1
  • Offline Offline
  • Posts: 56
    • View Profile
Re: Sia Official Nxt Thread
« Reply #54 on: July 09, 2014, 09:09:57 pm »

For an annual return of $10,000 per Sianote, $2.565b annualy would need to be spent over Sia. As a monopolist, this should be on the upper bound of achievable.

2,565,000,000 * 0.0000039 = $10,003.5

You made a mistake in your math somewhere.
Logged

Berry

  • Jr. Member
  • **
  • Karma: +12/-26
  • Offline Offline
  • Posts: 88
    • View Profile
Re: Sia Official Nxt Thread
« Reply #55 on: July 09, 2014, 09:27:03 pm »

For an annual return of $10,000 per Sianote, $2.565b annualy would need to be spent over Sia. As a monopolist, this should be on the upper bound of achievable.

2,565,000,000 * 0.0000039 = $10,003.5

You made a mistake in your math somewhere.


of course you are right.

Then easy game  ;D
Logged

Taek

  • Jr. Member
  • **
  • Karma: +6/-1
  • Offline Offline
  • Posts: 56
    • View Profile
Re: Sia Official Nxt Thread
« Reply #56 on: July 11, 2014, 10:36:12 pm »

What did I do today?

Today I spent most of the day looking at erasure coding. The problem that we discovered was that our original approach of using Cauchy Reed-Solomon codes was too bandwidth-intense when trying to recover files. Prohibitively expensive. Luckily, there are these fantastic things called Regenerating Codes, which are a more modern version of RS codes. I have no idea what type of patient minefield might be sprinkled around this, but it's mostly come from 1 set of researchers at Berkley, which makes me optimistic.

So, how do regenerating codes work? http://arxiv.org/pdf/1005.4178.pdf

Basically, you have 3 values that you care about: n, k, and d. N is the number of nodes holding fragments of a file. In our case, N=128 at all times. K is the number of fragments you need to recover the file. D is the number of partial fragments you need to do a low-bandwidth recovery of other fragments.

One unfortunate result seems to be that in order to do low-bandwidth recovery of fragments (absolutely mandatory), you need a minimum redundancy of 2. Adjusting for network volatility, the necessary redundancy approaches 2.5. So, the cost of having self-repairing storage on Sia is having a minimum redundancy of 2.5. Everything below that needs to be repaired manually because the bandwidth expenses of replacing a piece (need to rebuild the entire file) are too high.

I have no idea what the computational complexity of Regenerating Codes are, but it might be bad enough to require an ASIC to achieve network-speeds. Reed-Solomon Codes are already pretty and these are pretty much guaranteed to be slower.

A big concern is network growth. Every time the network grows, a few pieces get lost and need to be regenerated. This can be handled very efficiently if redundancy is high enough and friendly towards auto-regeneration. But if it's not, a network that's doubling in size every 2 weeks will wipe out every non-regenerating file in probably... 2 weeks.

Having such a high minimum redundancy is both good and bad. It's good because at a redundancy of 2.5 I can happily guarantee any file will be secure under anything but the most extreme circumstances. Even an attacker will need to control something like 30% of the network before they are able to target a single file. They'll need closer to 50% of the network before they can target a majority of files and 70-80% of the network before they are guaranteed to be able to target any file that auto-regenerates. It also means that files can be downloaded in more optimal ways; you only need to connect to 3/8 of the hosts to download your file, and you can pick the 3/8 that are cheapest/fastest/closest. It's a large optimization.

It's bad because 2.5x redundancy means you have to pay more. Originally I thought we could get away with a redundancy of 1.2. But it might be worth the tradeoff anyway. If we force all files to be at 2.5 redundancy minimum, we make it a lot less attractive for anyone to try and jockey for a position as an attacker. If you have files at 1.1 redundancy, a single party controlling 10% of the network is going to be able to attack around half of those files (corrupting them by deleting too many fragments to enable recovery). A single person at 10% isn't likely (Sia doesn't have mining pools), but it's a lot easier than 30%.

Would anyone be upset about the idea of a forced minimum redundancy? Fundamentally I feel against the idea, but there are some pretty clear advantages.
Logged

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Sia Official Nxt Thread
« Reply #57 on: July 11, 2014, 10:41:56 pm »

What did I do today?

Today I spent most of the day looking at erasure coding. The problem that we discovered was that our original approach of using Cauchy Reed-Solomon codes was too bandwidth-intense when trying to recover files. Prohibitively expensive. Luckily, there are these fantastic things called Regenerating Codes, which are a more modern version of RS codes. I have no idea what type of patient minefield might be sprinkled around this, but it's mostly come from 1 set of researchers at Berkley, which makes me optimistic.

So, how do regenerating codes work? http://arxiv.org/pdf/1005.4178.pdf

Basically, you have 3 values that you care about: n, k, and d. N is the number of nodes holding fragments of a file. In our case, N=128 at all times. K is the number of fragments you need to recover the file. D is the number of partial fragments you need to do a low-bandwidth recovery of other fragments.

One unfortunate result seems to be that in order to do low-bandwidth recovery of fragments (absolutely mandatory), you need a minimum redundancy of 2. Adjusting for network volatility, the necessary redundancy approaches 2.5. So, the cost of having self-repairing storage on Sia is having a minimum redundancy of 2.5. Everything below that needs to be repaired manually because the bandwidth expenses of replacing a piece (need to rebuild the entire file) are too high.

I have no idea what the computational complexity of Regenerating Codes are, but it might be bad enough to require an ASIC to achieve network-speeds. Reed-Solomon Codes are already pretty and these are pretty much guaranteed to be slower.

A big concern is network growth. Every time the network grows, a few pieces get lost and need to be regenerated. This can be handled very efficiently if redundancy is high enough and friendly towards auto-regeneration. But if it's not, a network that's doubling in size every 2 weeks will wipe out every non-regenerating file in probably... 2 weeks.

Having such a high minimum redundancy is both good and bad. It's good because at a redundancy of 2.5 I can happily guarantee any file will be secure under anything but the most extreme circumstances. Even an attacker will need to control something like 30% of the network before they are able to target a single file. They'll need closer to 50% of the network before they can target a majority of files and 70-80% of the network before they are guaranteed to be able to target any file that auto-regenerates. It also means that files can be downloaded in more optimal ways; you only need to connect to 3/8 of the hosts to download your file, and you can pick the 3/8 that are cheapest/fastest/closest. It's a large optimization.

It's bad because 2.5x redundancy means you have to pay more. Originally I thought we could get away with a redundancy of 1.2. But it might be worth the tradeoff anyway. If we force all files to be at 2.5 redundancy minimum, we make it a lot less attractive for anyone to try and jockey for a position as an attacker. If you have files at 1.1 redundancy, a single party controlling 10% of the network is going to be able to attack around half of those files (corrupting them by deleting too many fragments to enable recovery). A single person at 10% isn't likely (Sia doesn't have mining pools), but it's a lot easier than 30%.

Would anyone be upset about the idea of a forced minimum redundancy? Fundamentally I feel against the idea, but there are some pretty clear advantages.
Very interesting tradeoffs...
Anyway to make this a user selectable thing? people could have faster access and guaranteed availability, but they need 2.5 redundancy. If they are ok with having to spend time regenerating, then they can get away with lower cost. I dont see a one size fits all here.

James

P.S. Also assuming network size doubling every 2 weeks is quite unrealistic, unless you are staring from a network of 1 node :)
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

Vega

  • Full Member
  • ***
  • Karma: +8/-3
  • Offline Offline
  • Posts: 102
    • View Profile
Re: Sia Official Nxt Thread
« Reply #58 on: July 11, 2014, 10:57:17 pm »

I think raising the minimum redundancy is mainly a business decision, becase unless I'm mistaken, it raises the cost for anyone who wants to buy storage.

In my opinion the minimum redundancy needs to be at a level that makes data loss very unlikely. Because no matter how much you will explan people that they need to raise redundancy to make their files more secure, any data loss will result in bitching and moaning, and lost business.


Logged

Taek

  • Jr. Member
  • **
  • Karma: +6/-1
  • Offline Offline
  • Posts: 56
    • View Profile
Re: Sia Official Nxt Thread
« Reply #59 on: July 11, 2014, 11:46:11 pm »

Users would also have to spend money regenerating. Hosts are accountable for going offline, but only within reason. If you set redundancy to 1.2, then a host going offline will result in a requirement of having the whole original file rebuilt before the piece can be replaced. At 1.2 redundancy, that's something like 100x. (Need to download 100 complete pieces in order to rebuild the lost piece). At redundancy 2.5 you only need to download 2 pieces to rebuild the lost piece, which is a much friendlier number. The network would cover these 2 pieces for you but then you'd have to cover what remains. Given that hosts go offline all the time, (I'd expect in the ballpark of 1 per quorum per day), you'd need to spend more on recovery bandwidth then you'd be spending on storage. Overall I think 2.5 would be more cost effective.

But circumstances could be different for different files. If you're only planning on storing the file for 3 weeks, maybe you'd just do no repairing at all. There's a pretty strong chance that at 1.2 redundancy your file would be fine for 3 weeks. Or maybe you have a local machine that keeps a whole copy of the file for some reason (it's your OS or something like that) and you can do the repairing yourself without incurring the huge bandwidth hit.

I'd be worried about greedy users thinking that 1.2 is a safe value (or even 1.5) when without auto-repair, your file is a ticking time bomb. Then again, in the standard clients we could just hide the fact that the protocol supports non-repairing files. (and, a third party can always manage the repairs, the network doesn't have to).
Logged
Pages: 1 2 [3] 4 5 ... 8  All
 

elective-stereophonic
elective-stereophonic
assembly
assembly