Nxt Forum

Nxt Discussion => Nxt Core Development => Core Development Discussion => Topic started by: Jean-Luc on November 11, 2015, 05:14:14 pm

Title: Current 1.7 changelog
Post by: Jean-Luc on November 11, 2015, 05:14:14 pm
In the spirit of keeping the community informed about what is coming in 1.7, here is the changelog, as it is now. All features described are already implemented, this is not a roadmap, this is completed work (for the server part at least, we are still waiting for the UI to be finished and polished).


Coin Shuffling
This feature is based on the paper by Tim Ruffing et al, http://crypsys.mmci.uni-saarland.de/projects/CoinShuffle/coinshuffle.pdf .

Coin shuffling can be used to perform mixing of NXT, MS currencies (unless created as non-shuffleable), or AE assets. Any account can create a new shuffling, specifying the holding to be shuffled, the shuffle amount, number of participants required, and registration deadline. This is done using the shufflingCreate API. The subsequent shuffling steps can be done either manually, by using the shufflingRegister (for accounts other than the creator), shufflingProcess, shufflingVerify or shufflingCancel APIs, or, much more conveniently, by starting an automated Shuffler, using the startShuffler API. Once started, the Shuffler monitors the blockchain state for transactions relevant to the specified shuffle, and automatically submits the required transactions on behalf of the user, performing shuffle processing, verification, or cancellation as needed. To do this, the Shuffler is required to keep the user secret phrase in memory, therefore it should be run on a trusted local machine only. A restart or a crash of the node requires the shuffler to be started again using the startShuffler API, as it should never save the user secret phrase on disk.

To participate in a shuffling, a deposit of 1000 NXT is needed, in addition to the amount of currency or asset being shuffled. Or if shuffling NXT, the amount of the shuffle must exceed this 1000 NXT minimum. If the shuffling completes successfully, this amount is added to the recipient account balance, to allow it to send outgoing transactions (as it is required that only new, unused accounts are specified as recipients). If the shuffle fails due to a registered participant failing to participate as required, or intentionally submitting false data, the participant responsible for the shuffle cancellation is penalized by retaining this deposit and sending it to the forger of the shuffle finish block instead. If a shuffle is cancelled because the required number of participants is not met, nobody is penalized and all deposits are refunded.

Query APIs to retrieve currently running shufflers, shufflings, and shuffling participants are: getAllShufflings, getAccountShufflings, getAssignedShufflings, getHoldingShufflings, getShufflers, getShuffling, and getShufflingParticipants.

If desired, finished shufflings can be automatically deleted from the database if the nxt.deleteFinishedShufflings property is set to true (default is false).

The fee for creating a shuffling or registering in one is 1 NXT, for the shuffling process or shuffling cancel transactions 10 NXT, and for the verify transaction 1 NXT.


Account control for phased transactions
Any account can be restricted to only be allowed to issue phased transactions subject to a specific voting model. This is achieved by the account submitting a setPhasingOnly transaction using the setPhasingOnlyControl API. The getPhasingOnlyControl API can be used to retrieve the status of an account phasing control, and getAllPhasingOnlyControls to get all accounts subject to phasing control with their respective restrictions.

Once set, the phasing only account control can only be disabled or changed with another setPhasingOnly transaction, itself subject to the currently set phasing restrictions.

Note that by-transaction and by-hash voting models are not allowed for phasing control, and setting voting model to none is used to disable the control.


Immediate release of phased transactions on approval
Phased transactions with a voting model that does not depend on account balance (such as by-transaction or by-hash), or by-account with no minimum balance and with a whitelist, will be released before their finish height as soon as approved (in the block in which the transaction causing their approval is executed), if possible. Such early finish is guaranteed for transaction types known to be phasing safe. For others, if the early finish does not succeed due to the transaction failing validation at this height or conflicting with another transaction in the same block, a second, final release attempt will be performed at finish height.


New base target adjustment algorithm
Average block times will be 60 s, with 1440 blocks per day. Block times should practically never exceed 10 min.

Limit of 1000 NXT on minimum forging balance. This applies to the total of the account own guaranteed balance plus any balances leased to it, but not to each individual balance lease. An account with balance lower than the limit can still lease its balance to another.


Account properties
Those are name / value pairs that can be set on any account (except Genesis), by either the account owner, or by another account. Names are limited to 32 characters, and values to 160 characters. Names are unique per account and per setter account, but not globally unique. Account properties cannot be transferred between accounts. The setter of an account property can edit it by replacing its value with another. Either the setter, or the recipient (if different) of an account property can delete it. There is no limit on the number of properties an account can have. Fee for setting account property is 1 NXT for value up to 32 chars, with additional 1 NXT fee for every 32 chars after that.

Account properties are managed using the setAccountProperty and deleteAccountProperty APIs. To query the properties of an account, or those set by an account, the getAccountProperties API can be used.


Singleton assets
Issuing an asset with a quantity of 1, decimals 0, and description length not exceeding 160 characters, will require a base minimum fee of 1 NXT only, instead of the regular 1000 NXT asset issuance fee. For description of more than 32 chars, an extra 1 NXT fee is added for each 32 chars. Asset name for singleton assets is limited to 10 chars, same as for regular assets.


Throttling of unique resource allocation transactions
Asset issuance (excluding singleton assets), monetary system currency issuance, and alias assignment (excluding re-assignment), will be limited to only one transaction of each type accepted per block.


Spreading back block fees for asset and currency issuance
The transaction fees for asset (excluding singleton assets) and currency issuance will be split between the forgers of the current and the previous three blocks in a 4:3:2:1 ratio.


Deletion of asset shares will be performed as a separate AssetDelete transaction type instead of as sending the shares to Genesis. Sending shares to Genesis will no longer be allowed.


Fees and size limit changes

Several transaction types or attachments will have new fees and size limits, to encourage users to utilize the prunable versions when available, and to make fees proportionate to actual blockchain space consumed.

Aliases:
Base fee 2 NXT, with 2 NXT additional fee for each 32 chars of name plus URI total length, after the first 32 chars. Name and URI size limits remain at 100 and 1000 chars respectively.

Messages and EncryptedMessages (non-prunable):
Maximum length reduced to 160 bytes. 1 NXT fee for each 32 bytes after the first 32 bytes. For encrypted messages, the length is measured excluding the nonce and the 16 byte AES initialization vector.

Fees and size limit for prunable messages remain unchanged.

AccountInfo:
Base fee 1 NXT, with 2 NXT additional fee for each 32 chars of name plus description total length, after the first 32 chars. Name and description size limits remain at 100 and 1000 chars. AccountInfo transactions throttled at one per block.

Polls:
Base fee 10 NXT for polls with up to 20 options, and total size of poll name plus poll description plus total option length not exceeding 320 chars. For each option above 20, an additional fee of 1 NXT, and for each 32 chars after 320, an additional fee of 1 NXT. Poll creation throttled to one per block. Name, description, and option length limits remain at 100, 1000, and 100 chars respectively.

DGS Listing:
Base fee 2 NXT, with 2 NXT additional fee for each 32 chars of name plus description total length, after the first 32 chars. Name and description size limits remain at 100 and 1000 max. DGS Listing throttled at one per block.

DGS Delivery:
Base fee 1 NXT, with 2 NXT additional fee for each 32 bytes of encrypted goods data after the first 32 bytes, nonce and AES initialization bytes excluded. Encrypted goods data size limit remains 1000 bytes.


Title: Re: Current 1.7 changelog
Post by: Sebastien256 on November 11, 2015, 05:15:36 pm
Thanks you JL, great post!
Title: Re: Current 1.7 changelog
Post by: _mr_e on November 11, 2015, 06:08:32 pm
Who has reviewed the shuffle system? Does this mean we should be able to deposit btc into mgw and shuffle the superbtc, effectively creating a super cheap bitcoin mixer?
Title: Re: Current 1.7 changelog
Post by: Ore Mountains on November 11, 2015, 06:12:39 pm
Cool, also thanks for linking the paper!
Title: Re: Current 1.7 changelog
Post by: EvilDave on November 11, 2015, 06:21:25 pm
Who has reviewed the shuffle system? Does this mean we should be able to deposit btc into mgw and shuffle the superbtc, effectively creating a super cheap bitcoin mixer?

Tim Ruffing did the review.....
Here's his original paper on coin shuffling again:
https://www.petsymposium.org/2014/papers/Ruffing.pdf
Title: Re: Current 1.7 changelog
Post by: _mr_e on November 11, 2015, 06:24:12 pm
Who has reviewed the shuffle system? Does this mean we should be able to deposit btc into mgw and shuffle the superbtc, effectively creating a super cheap bitcoin mixer?

Tim Ruffing did the review.....
Here's his original paper on coin shuffling again:
https://www.petsymposium.org/2014/papers/Ruffing.pdf

Awesome, I saw a forum post a while back where he was critiquing some issues, I never saw a follow up that he had okd the implementation. but that's great if so! Do you have a link if it was publicly discussed?
Title: Re: Current 1.7 changelog
Post by: farl4bit on November 11, 2015, 06:34:52 pm
What a great changelog. Although I don't understand all of it, it looks really promising. How is it going with the development? Hard work? Allnighters?
Title: Re: Current 1.7 changelog
Post by: bidji29 on November 11, 2015, 07:03:05 pm
What is the reason for limiting those special transaction at 1/block?
I guess it's to limit the abuse and bloat but 1 is very low.
WHy not make it like 3 or 5, it would cover our base for longer in case the use of NXT increase
Title: Re: Current 1.7 changelog
Post by: Seccour on November 11, 2015, 07:08:04 pm
" Throttling of unique resource allocation transactions
Asset issuance (excluding singleton assets), monetary system currency issuance, and alias assignment (excluding re-assignment), will be limited to only one transaction of each type accepted per block. "

Why do you want to limit it ?
Title: Re: Current 1.7 changelog
Post by: Jean-Luc on November 11, 2015, 07:19:21 pm
The reason for throttling was discussed before, it is one of the ways to protect against forgers acquiring unique resources for free. The other is the spreading back of fees for such transactions.

Just like fees, the throttling limit can be changed at the next hard fork if needed. We don't get a sustained flow of 60 new assets per hour, or currencies, or aliases. And throttling just means your transaction gets to wait until the next block. If in a hurry, sending it with a higher fee will make sure it gets in a block faster.
Title: Re: Current 1.7 changelog
Post by: kushti on November 11, 2015, 07:27:28 pm
Is some help with tests writing / code reviewing needed? Please PM me if so
Title: Re: Current 1.7 changelog
Post by: allwelder on November 11, 2015, 11:59:16 pm
Thanks.


Singleton assets
Issuing an asset with a quantity of 1, decimals 0, and description length not exceeding 160 characters, will require a base minimum fee of 1 NXT only, instead of the regular 1000 NXT asset issuance fee. For description of more than 32 chars, an extra 1 NXT fee is added for each 32 chars. Asset name for singleton assets is limited to 10 chars, same as for regular assets.

What's the purpose of this feature?
Title: Re: Current 1.7 changelog
Post by: Sabertooth on November 12, 2015, 12:14:37 am
Thanks.


Singleton assets
Issuing an asset with a quantity of 1, decimals 0, and description length not exceeding 160 characters, will require a base minimum fee of 1 NXT only, instead of the regular 1000 NXT asset issuance fee. For description of more than 32 chars, an extra 1 NXT fee is added for each 32 chars. Asset name for singleton assets is limited to 10 chars, same as for regular assets.

What's the purpose of this feature?

Use your imagination, gold bars with serial numbers etc....
Title: Re: Current 1.7 changelog
Post by: allbits on November 12, 2015, 01:41:52 am
Use cases for singleton assets? - How about the issuance of negotiable instruments?

https://en.wikipedia.org/wiki/Negotiable_instrument

Let's say twinwinnerd wants to borrow money from me?  I lend him the money, on certain terms, and he sends me a singleton asset that describes these terms.  Upon the expiry of a certain defined period, the bearer of the asset can present to twinwinnerd and he will honor the obligation (or default, as the case may be).

During the time period in question, I might choose to hold the singleton asset or transfer it to another person by listing it on the AE or on the DGS or on FreeMarket.  In the middle ages, bills of exchange traversed the continent from one holder to another, at various discount rates.  Of course, one main purpose for this was to span space and time, and therefore my example may be merely a quaint reference to a former time rather than an actual use case.

Or perhaps asset issuers might decide to accept "Bills of Exchange" in exchange for the issuance of assets, for example, if they trusted the creditor in question.  They then might be able to sell the Bill of Exchange on the marketplace for immediate cash sometime later, provided they offered a satisfactory discount rate.  Of course, the creditor could easily do this directly but, who knows, such an exchange might make sense for all the parties involved.

Or perhaps an asset owner of an asset that is being actively traded might wish to lend stock to a market maker who wants to take short positions (or to allow others to take short positions).  They could agree on detailed terms about time to repay, form of repayment and compensation/interest.  The stock lender would then have a negotiable instrument that, perhaps, he could sell prior to expiration if the need arose or if he wanted to exit the position at a favorable time prior to expiration of the instrument.
Title: Re: Current 1.7 changelog
Post by: qq2536007339 on November 12, 2015, 01:52:11 am
What's a great changelog.
Title: Re: Current 1.7 changelog
Post by: allbits on November 12, 2015, 04:41:20 am
Also, bearer bonds are a use case for singleton assets

https://en.wikipedia.org/wiki/Bearer_bond

Supernet, for example, could issue bearer bonds, with a set face value, that pay interest or dividends of some kind for a period of time.  These bonds could then be freely tradable in an after market.
Title: Re: Current 1.7 changelog
Post by: Sebastien256 on November 12, 2015, 06:57:14 am
Thanks.


Singleton assets
Issuing an asset with a quantity of 1, decimals 0, and description length not exceeding 160 characters, will require a base minimum fee of 1 NXT only, instead of the regular 1000 NXT asset issuance fee. For description of more than 32 chars, an extra 1 NXT fee is added for each 32 chars. Asset name for singleton assets is limited to 10 chars, same as for regular assets.

What's the purpose of this feature?
one possibility:
https://nxtforum.org/core-development-discussion/list-of-feature-request-for-nrs/msg184211/#msg184211

I think this feature will be use for the mmo voxelnaults
Title: Re: Current 1.7 changelog
Post by: bcdev on November 12, 2015, 08:33:20 am
Thanks.


Singleton assets
Issuing an asset with a quantity of 1, decimals 0, and description length not exceeding 160 characters, will require a base minimum fee of 1 NXT only, instead of the regular 1000 NXT asset issuance fee. For description of more than 32 chars, an extra 1 NXT fee is added for each 32 chars. Asset name for singleton assets is limited to 10 chars, same as for regular assets.

What's the purpose of this feature?
Imagine a car that opens only to a person who has the token.
Then buying/selling that car is literally "buy a token on AE" and drive to it's location. The car will recognize you automatically as a new owner.
Title: Re: Current 1.7 changelog
Post by: coinomat on November 12, 2015, 08:46:17 am
Lot's of action could come out of this.
asset deletion is a nice feature.
To be frank I'm not really worried about coin shuffling any more, in the short run it just won't have sufficient volume to really matter
Title: Re: Current 1.7 changelog
Post by: cc001 on November 12, 2015, 08:56:30 am
To be frank I'm not really worried about coin shuffling any more, in the short run it just won't have sufficient volume to really matter
Why are you worried about coin shuffling (or have been in the beginning)? I think it is absolutely crucial for every cryptocurrency (including BTC) to be completely anonymous to become mainstream (and I mean regular use cases, not blackmarket stuff).
It is a GREAT step forward for Nxt to implement that protocol, we should promote/market that heavily
Title: Re: Current 1.7 changelog
Post by: Isildur23 on November 12, 2015, 08:57:01 am
Wow! Great changes! Thank you!
Title: Re: Current 1.7 changelog
Post by: coinomat on November 12, 2015, 09:58:13 am
To be frank I'm not really worried about coin shuffling any more, in the short run it just won't have sufficient volume to really matter
Why are you worried about coin shuffling (or have been in the beginning)? I think it is absolutely crucial for every cryptocurrency (including BTC) to be completely anonymous to become mainstream (and I mean regular use cases, not blackmarket stuff).
It is a GREAT step forward for Nxt to implement that protocol, we should promote/market that heavily
I have to start a separate thread about it probably. too many questions about this. on the other hand at the moment I don't want to do anything that could potentially hurt NXT
so let this be implemented first.
Title: Re: Current 1.7 changelog
Post by: Ludom on November 12, 2015, 02:04:46 pm
Shuffling is not about anonymity, it's about privacy. You will always see who owns what. The ownership is always visible, it changes only for payment.

Every thing is transparent, excepted some transactions. But for that, you can already shuffle outside the Nxt system with centralised systems. It's great that Nxt proposes an inside and trustless solution.
Title: Re: Current 1.7 changelog
Post by: Damelon on November 12, 2015, 04:16:17 pm
Shuffling is not about anonymity, it's about privacy.

Thanks for this very important difference.

Privacy is a basic civil right.

Anonymity is a choice.

There is a big difference between the two.
Once people who insist on their basic rights are treated as criminals or prospective criminals, we are on a slippery slope.

Title: Re: Current 1.7 changelog
Post by: BTCDDev on November 12, 2015, 04:42:16 pm
What is the status of the UI?

Is there a timeframe?
Title: Re: Current 1.7 changelog
Post by: Jean-Luc on November 12, 2015, 05:26:50 pm
Who has reviewed the shuffle system? Does this mean we should be able to deposit btc into mgw and shuffle the superbtc, effectively creating a super cheap bitcoin mixer?

Tim Ruffing did the review.....
Here's his original paper on coin shuffling again:
https://www.petsymposium.org/2014/papers/Ruffing.pdf

Yes, and here is my discussion with Tim:
https://nxtforum.org/core-development-discussion/coin-shuffling-code-review-by-tim-ruffing/
Title: Re: Current 1.7 changelog
Post by: LibertyNow on November 14, 2015, 05:41:44 pm
Really fantastic change log.  A lot of exciting things coming down the pipe!  Good work @Jean-Luc and other developers.  This is why I'm doing the LQD IPO and why I continue to invest time and resources into NXT.  The tech is top tier and the best option for businesses. Once the crypto community gets this, we will see a massive new flow of funding.

LibertyNow
Title: Re: Current 1.7 changelog
Post by: chanc3r on November 14, 2015, 09:19:15 pm
Quote

Messages and EncryptedMessages (non-prunable):
Maximum length reduced to 160 bytes. 1 NXT fee for each 32 bytes after the first 32 bytes. For encrypted messages, the length is measured excluding the nonce and the 16 byte AES initialization vector.

I'm posting this here and in the other threads so its on record...
A Maximum permanent message length of 160 will cause some MGW transactions to fail unless this MGW is changed or this length is increased, still figuring out the minimum needed to sustain MGW unchanged but 160 is definitely too low. This will also increase MGW transaction costs depending on the base fee - for those of you voting...

BTW I already told J-L this but after this discussion was created.
Title: Re: Current 1.7 changelog
Post by: Jean-Luc on November 14, 2015, 09:27:11 pm
The MGW must switch to prunable, and use archival nodes to retrieve expired prunable messages if needed. This is the solution, not forcing every node to store MGW messages forever, that only those using MGW need.
Title: Re: Current 1.7 changelog
Post by: jl777 on November 15, 2015, 12:15:00 am
The MGW must switch to prunable, and use archival nodes to retrieve expired prunable messages if needed. This is the solution, not forcing every node to store MGW messages forever, that only those using MGW need.
As I posted elsewhere, I do not see how to make MGW reliable while using prunable messages, so any fee increase will have to be passed onto users.

With prunable messages, the edge case of an already paid deposit being forgotten as the data needed to make that correlation gets pruned away and cannot be found, will exist. That means the possibility of multiple payments sent out of the MGW and to prevent that would require all sorts of extra work. Also, the withdraws from users could also be forgotten, but that is less likely of a scenario as long as MGW doesnt go down for 2 weeks.

The deposit transfer though, that is something that is rescanned from scratch each time the MGW restarts. So if the data is pruned, it will look like a pending deposit that hasnt been paid.

Keep in mind these are NXT transactions, not MGW messages, as each MGW message is part of a transaction.

James
Title: Re: Current 1.7 changelog
Post by: Jean-Luc on November 15, 2015, 06:48:46 am
As I posted elsewhere, I do not see how to make MGW reliable while using prunable messages, so any fee increase will have to be passed onto users.
I already explained elsewhere how to use prunable data and how they are as reliable as permanent.
Title: Re: Current 1.7 changelog
Post by: blackyblack1 on November 15, 2015, 06:57:32 am
The MGW must switch to prunable, and use archival nodes to retrieve expired prunable messages if needed. This is the solution, not forcing every node to store MGW messages forever, that only those using MGW need.
As I posted elsewhere, I do not see how to make MGW reliable while using prunable messages, so any fee increase will have to be passed onto users.

With prunable messages, the edge case of an already paid deposit being forgotten as the data needed to make that correlation gets pruned away and cannot be found, will exist. That means the possibility of multiple payments sent out of the MGW and to prevent that would require all sorts of extra work. Also, the withdraws from users could also be forgotten, but that is less likely of a scenario as long as MGW doesnt go down for 2 weeks.

The deposit transfer though, that is something that is rescanned from scratch each time the MGW restarts. So if the data is pruned, it will look like a pending deposit that hasnt been paid.

Keep in mind these are NXT transactions, not MGW messages, as each MGW message is part of a transaction.

James
You can simply switch off prunning on your MGW machines. It will work the same way as permanent messages unless all your data is lost in some catastrophe.
Title: Re: Current 1.7 changelog
Post by: jl777 on November 15, 2015, 07:07:45 am
The MGW must switch to prunable, and use archival nodes to retrieve expired prunable messages if needed. This is the solution, not forcing every node to store MGW messages forever, that only those using MGW need.
As I posted elsewhere, I do not see how to make MGW reliable while using prunable messages, so any fee increase will have to be passed onto users.

With prunable messages, the edge case of an already paid deposit being forgotten as the data needed to make that correlation gets pruned away and cannot be found, will exist. That means the possibility of multiple payments sent out of the MGW and to prevent that would require all sorts of extra work. Also, the withdraws from users could also be forgotten, but that is less likely of a scenario as long as MGW doesnt go down for 2 weeks.

The deposit transfer though, that is something that is rescanned from scratch each time the MGW restarts. So if the data is pruned, it will look like a pending deposit that hasnt been paid.

Keep in mind these are NXT transactions, not MGW messages, as each MGW message is part of a transaction.

James
You can simply switch off prunning on your MGW machines. It will work the same way as permanent messages unless all your data is lost in some catastrophe.
and its ok to just lose customer data in such a catastrophe?

and there wont ever be any migration to a new server, that maybe is misconfigured and doesnt have the data and then the startup sequence gets confused about phantom pending deposits?

So far we have had bitcoind corrupting its DB, data center breaking the relay nodes, and that is just recent history. In the real world things like that happen

James
Title: Re: Current 1.7 changelog
Post by: jl777 on November 15, 2015, 07:11:47 am
As I posted elsewhere, I do not see how to make MGW reliable while using prunable messages, so any fee increase will have to be passed onto users.
I already explained elsewhere how to use prunable data and how they are as reliable as permanent.
Are you really claiming that having the mgw data permanently in the blockchain with each mgwtx in the NXT blockchain is just as reliable as prunable data that is stored on whatever public nodes decide to keep them around and the local copies in the MGW servers?

so having 3 + P copies is as reliable as all NXT nodes and not having any special case code for finding the pruned data is just as simple and reliable as having code that checks locally, then remotely over the network to public nodes.

I am quite amazed by this

James
Title: Re: Current 1.7 changelog
Post by: Sabertooth on November 15, 2015, 07:18:31 am
As I posted elsewhere, I do not see how to make MGW reliable while using prunable messages, so any fee increase will have to be passed onto users.
I already explained elsewhere how to use prunable data and how they are as reliable as permanent.
Are you really claiming that having the mgw data permanently in the blockchain with each mgwtx in the NXT blockchain is just as reliable as prunable data that is stored on whatever public nodes decide to keep them around and the local copies in the MGW servers?

so having 3 + P copies is as reliable as all NXT nodes and not having any special case code for finding the pruned data is just as simple and reliable as having code that checks locally, then remotely over the network to public nodes.

I am quite amazed by this

James

I'm sure how many full blockchain copies that will be available will greatly outnumber the multisig data sets that you rely on to run the MGW servers.
Title: Re: Current 1.7 changelog
Post by: Jean-Luc on November 15, 2015, 09:08:18 am
There is no need to write any special case code to handle pruned data. Once your local servers are configured to store prunable data indefinitely, they take care of retrieving any missing prunable data from public archival nodes automatically. This was a major feature implemented in 1.6 (thanks, ScripterRon!). And such retrieval is only needed during the initial blockchain download, or if a server has been offline for more than 2 weeks.

I have also increased the default prunable lifetime from 2 weeks to 90 days in 1.6, so most nodes, public or not, will store 3 months worth of prunable data (although those running on limited performance devices can still prefer to reduce that to 2 weeks only). This gives you yet another redundancy layer, even if your servers are down for more than 2 weeks, and there are no public archival nodes, the last 3 months of data are still stored on almost all nodes.
Title: Re: Current 1.7 changelog
Post by: chanc3r on November 15, 2015, 09:36:49 am
I'm talking to other members of the team as well.

MGW servers are already set to store prunable messages because of the risk the transfer requests coming in from client users are prunable.

As I understand it when a node runs the archive service it simply stores the blockchain 'unabridged' and the API calls work as expected, so if the node has the non pruned blocks then it will return it in the API call, nothing special needs to be done.

Basically NXT has evolved to have two types of peer, some storing the unabridged blockchain and some storing the pruned chain. Most end users (not using a lite client) will run a peer which retrieves the unabridged blockchain and it can get this from either type of peer.

Services like MGW & probably other businesses needing such records will need to run a peer which implements the archive service which will need to contact and retrieve the full blockchain from other peers also running the archive service..

This is where I think some nervousness comes in - does anyone know how many peers there are out there which are running the archive service ?
How can I be sure that I can always find an archive peer?
If at any point there are no archive peers running then the full blockchain cannot be retrieved if I need a new download.

Knowing and being confident of the size of the community you can rely on for that back up the full blockchain that you need I think is an important asoect of this change to 1.7 which will drive the use of prunable data.
Title: Re: Current 1.7 changelog
Post by: Sebastien256 on November 15, 2015, 09:50:06 am
So, if I understand from chanc3r post, it would be be nice to have a place in the NRS client that show:
1) the number of node runing nxt client
2) the type of node, archive node or not.

That would be important and pertinent information to retrieve and show in the default NRS clients, imo. That would increase confidence in the retrievability of the Nxt prunable system.
Title: Re: Current 1.7 changelog
Post by: Seccour on November 15, 2015, 01:46:53 pm
The MGW must switch to prunable, and use archival nodes to retrieve expired prunable messages if needed. This is the solution, not forcing every node to store MGW messages forever, that only those using MGW need.

Can we not just wait until MGW devs find a way to switch to prunable or to only use 32 bytes ?
Title: Re: Current 1.7 changelog
Post by: Brangdon on November 15, 2015, 03:30:05 pm
Can we not just wait until MGW devs find a way to switch to prunable or to only use 32 bytes ?
Jean-Luc already offered to delay the new fee block; this can be done without delaying the other new features. He'd need to know how long for.

Increasing the maximum from 160 to 200 bytes might also be possible; Jean-Luc doesn't seem to want to, but I've not seen any rationale for why 160 bytes is the magic number here, rather than 128 or 256. Currently the vote seems to favour 2 NXT basic fee, so 200 bytes would cost 14 NXT. Of course, the next rounds of voting might push the cost up to 21 NXT or 28 NXT. That would provide SuperNet with some incentive to switch to prunable data, so after a while hopefully few transactions would actually be using the extra 40 bytes. Maybe the limit could be dropped again in a year's time, if nobody then needs it.
Title: Re: Current 1.7 changelog
Post by: _mr_e on November 15, 2015, 03:35:09 pm
Could we please add a setting that when enabled would automatically show unsigned bytes with the QR code rather then forcing you to click, "do not broadcast" & "do not sign" when the passphrase is not available?
Title: Re: Current 1.7 changelog
Post by: jl777 on November 15, 2015, 03:51:08 pm
Can we not just wait until MGW devs find a way to switch to prunable or to only use 32 bytes ?
Jean-Luc already offered to delay the new fee block; this can be done without delaying the other new features. He'd need to know how long for.

Increasing the maximum from 160 to 200 bytes might also be possible; Jean-Luc doesn't seem to want to, but I've not seen any rationale for why 160 bytes is the magic number here, rather than 128 or 256. Currently the vote seems to favour 2 NXT basic fee, so 200 bytes would cost 14 NXT. Of course, the next rounds of voting might push the cost up to 21 NXT or 28 NXT. That would provide SuperNet with some incentive to switch to prunable data, so after a while hopefully few transactions would actually be using the extra 40 bytes. Maybe the limit could be dropped again in a year's time, if nobody then needs it.
the minimum size that can be used to correlate a BTC deposit would be the txid + vout, so that is 32 bytes of high entropy + integer, 34 bytes takes care of most cases, but if BTC blocks get bigger and we see more than 65536 vouts in a tx, 35 bytes would be needed.

However, this requires more than just server settings changes and would require a new round of testing. in any case I cannot fit 35 bytes into 32, so it is a moot point as it has been declared that 32 bytes is the limit. Also this assumes only BTC deposits, so all the coins would need to be scanned and what if there is a txid collision between blockchains? It is possible and actually happened in the early btc blockchain, so add another byte for the coin. maybe there will be some other things that are nice to have, like the NXT address of 8 bytes. some coin's txid dont fit into 32 bytes, etc.

As far as how long, this is  a question for vanbreuk and the new MGW dev

James
Title: Re: Current 1.7 changelog
Post by: jl777 on November 15, 2015, 04:00:37 pm
BTC txfee is .0001 -> ~3 cents and you can put at least 80 bytes in OP_RETURN on the BTC blockchain.

would you pay more to store data in the NXT blockchain vs the BTC blockchain?

economics

price elasticity

these are not hard to understand. expensive things have less volumes than less expensive things.

If the cost to use NXT storage is the most expensive in all of crypto, then it will probably become the least used blockchain storage. Basically unless the effective cost to use NXT is significantly lower than BTC, new use cases wont use NXT. And if it is more expensive, current use cases will be economically forced to migrate

There has been 2 years for NXT ecosystem to develop around the 1NXT per tx and 1000 bytes pricing. If the cost goes up, the usage will go down. If the cost goes down, the usage might go up.

Why not make it cost 1000 NXT per tx and another 100NXT per byte used? that will make 1000x the forging income and everybody will want to forge NXT

James
Title: Re: Current 1.7 changelog
Post by: Brangdon on November 15, 2015, 04:26:21 pm
the minimum size that can be used to correlate a BTC deposit would be the txid + vout, so that is 32 bytes of high entropy + integer, 34 bytes takes care of most cases, but if BTC blocks get bigger and we see more than 65536 vouts in a tx, 35 bytes would be needed.

However, this requires more than just server settings changes and would require a new round of testing. in any case I cannot fit 35 bytes into 32, so it is a moot point as it has been declared that 32 bytes is the limit.
As I understand it, the limit will be 160 bytes. 32 bytes is what you get for the base fee. You can get more space if you pay a higher fee. From the first post in this thread:

Messages and EncryptedMessages (non-prunable):
Maximum length reduced to 160 bytes. 1 NXT fee for each 32 bytes after the first 32 bytes.

If you can fit your stuff into 64 bytes, there's no problem using permanent storage other than having to pay for the second 32 bytes.

Maybe instead of txid you could use the block-height of the transaction plus an index into the block, or some other indexing scheme, to get under 32 bytes. It'll be more work, of course, but might be worth it if you hate prunable data that much, and want to pay the minimum fees. Really the point of the new fee structure is to encourage more efficient use of space in the Nxt blockchain.
Title: Re: Current 1.7 changelog
Post by: Jean-Luc on November 16, 2015, 08:22:57 am
Can we not just wait until MGW devs find a way to switch to prunable or to only use 32 bytes ?
Jean-Luc already offered to delay the new fee block; this can be done without delaying the other new features. He'd need to know how long for.

Increasing the maximum from 160 to 200 bytes might also be possible; Jean-Luc doesn't seem to want to, but I've not seen any rationale for why 160 bytes is the magic number here, rather than 128 or 256. Currently the vote seems to favour 2 NXT basic fee, so 200 bytes would cost 14 NXT. Of course, the next rounds of voting might push the cost up to 21 NXT or 28 NXT. That would provide SuperNet with some incentive to switch to prunable data, so after a while hopefully few transactions would actually be using the extra 40 bytes. Maybe the limit could be dropped again in a year's time, if nobody then needs it.

160 is a magic number indeed. I was initially thinking of 128 only, then decided to make it 32 for free, plus 128 paid, 160 total.

Changing this limit is not something we can do often, if at all. It must be done now, because 1000 is too high, and the only reason the limit was 1000 before is that we didn't have the prunable feature at that time. But we cannot set the limit to 256 now, and decide to lower it to 128 in the future, this will break client applications. Yes, the current change also breaks them, but it is unavoidable. If we must break them, do it once, and then no more changes in the limit. This is why I would rather delay setting the limit by a few weeks, than set it too high now with an intention to lower it more in the future.

The minimum transaction size now is 176 bytes, for ordinary payment with no attachment. 32 bytes of it is the sender public key with is stored only once per account, so the actual data that such a transaction adds is 144 bytes. Must be kept in mind that when expressed as JSON, to be sent over the network, the real size is about twice as much, and when saved in the database again there is overhead, such size measures are only approximate and not an exact indication of how much disk space or bandwith a transaction will consume.
Title: Re: Current 1.7 changelog
Post by: cc001 on November 16, 2015, 10:05:03 am
Account properties
Those are name / value pairs that can be set on any account (except Genesis), by either the account owner, or by another account. Names are limited to 32 characters, and values to 160 characters. Names are unique per account and per setter account, but not globally unique. Account properties cannot be transferred between accounts. The setter of an account property can edit it by replacing its value with another. Either the setter, or the recipient (if different) of an account property can delete it. There is no limit on the number of properties an account can have. Fee for setting account property is 1 NXT for value up to 32 chars, with additional 1 NXT fee for every 32 chars after that.

What are the use cases for those properties? What was the idea behind them? How can services use them? How are they useful for the regular user with the standard client?
Title: Re: Current 1.7 changelog
Post by: Brangdon on November 16, 2015, 11:48:22 am
160 is a magic number indeed. I was initially thinking of 128 only, then decided to make it 32 for free, plus 128 paid, 160 total.
Thanks for explaining.

Quote
But we cannot set the limit to 256 now, and decide to lower it to 128 in the future, this will break client applications.
My thought is that we could tell, from inspecting the block-chain, how much data client apps were actually using. If we reach a point where no app in the last N months has used more than 64 bytes, then we could drop the limit to 128 bytes and be reasonably sure of not breaking them. Technically it is a breaking API change, but one that would not inconvenience anyone in practice. We would also announce now that we planned to do this, so client developers would know that the larger attachments were deprecated and have fair warning.

That said, if the minimum transaction size is 176 bytes, then there are diminishing returns in limiting the attachment size much below that. Someone who actively wants to cause bloat, will get more bloat for their NXT by creating many messages at base cost and size, then by creating fewer with larger attachments. For 5 NXT you could get 5 messages with 32-byte attachments at 208 bytes each = 1040 bytes, or 1 message with 160-byte attachment for 336 bytes total. For this reason it seems to me that allowing larger attachments is actually safe, provided the cost-per-byte is high as planned. Perhaps the limit of 1000 bytes was not wrong; the problem was getting all those bytes for free.

To put it another way, I suspect SuperNet will be screwed by the high fees even if they aren't screwed by a lower limit. Either way, to have a viable service they will need to reduce the size of their permanent attachments.
Title: Re: Current 1.7 changelog
Post by: BTCDDev on November 23, 2015, 04:10:19 pm
Will it be possible for each unit of a Singleton asset to be assigned a unique identifier?
Title: Re: Current 1.7 changelog
Post by: Jean-Luc on November 23, 2015, 04:15:28 pm
The asset id of each is unique, but manually assign some other identifier, no, there is no field for that.
Title: Re: Current 1.7 changelog
Post by: BTCDDev on November 23, 2015, 04:27:59 pm
The asset id of each is unique, but manually assign some other identifier, no, there is no field for that.

Ok. I'm not so familiar with MS, is it possible to do that there?

To issue an asset/currency with each individual unit assigned some uid
Title: Re: Current 1.7 changelog
Post by: Jean-Luc on November 23, 2015, 05:30:46 pm
No, there is no way to track units of assets, currencies, or NXT itself. All those are stored as a single number for each, representing the account balance at a given blockchain height, but without possibility to track which coin/asset share came from which transaction.

The closest to what you ask for would indeed be singleton assets, if you assume some convention (not enforced by the core) that all singleton assets issued by a given account and having the same name, represent units of one "asset". However, you could not for example bundle those and put an ask/sell order for a few at a time, the core will still consider each a separate asset.
Title: Re: Current 1.7 changelog
Post by: Damelon on November 24, 2015, 12:39:55 pm
Account properties
Those are name / value pairs that can be set on any account (except Genesis), by either the account owner, or by another account. Names are limited to 32 characters, and values to 160 characters. Names are unique per account and per setter account, but not globally unique. Account properties cannot be transferred between accounts. The setter of an account property can edit it by replacing its value with another. Either the setter, or the recipient (if different) of an account property can delete it. There is no limit on the number of properties an account can have. Fee for setting account property is 1 NXT for value up to 32 chars, with additional 1 NXT fee for every 32 chars after that.

What are the use cases for those properties? What was the idea behind them? How can services use them? How are they useful for the regular user with the standard client?

I am also looking into these.

Can anyone shed some light on this?
Title: Re: Current 1.7 changelog
Post by: abctc on November 24, 2015, 01:49:25 pm
Account properties
Those are name / value pairs that can be set on any account (except Genesis), by either the account owner, or by another account. Names are limited to 32 characters, and values to 160 characters. Names are unique per account and per setter account, but not globally unique. Account properties cannot be transferred between accounts. The setter of an account property can edit it by replacing its value with another. Either the setter, or the recipient (if different) of an account property can delete it. There is no limit on the number of properties an account can have. Fee for setting account property is 1 NXT for value up to 32 chars, with additional 1 NXT fee for every 32 chars after that.
What are the use cases for those properties? What was the idea behind them? How can services use them? How are they useful for the regular user with the standard client?
I am also looking into these.
Can anyone shed some light on this?
- I think with Account properties and with Account Control (including Phased Txs) it would be possible to make trustless (and even website-less) equivalent of such service as tips.nxtex.info

(http://i73.fastpic.ru/big/2015/1124/14/b472914b90bd61c871fcd1dde7a3d214.png)

Just put that profile properties (size of donations, hash of PIN code, tips cookies) into your Account properties.
Title: Re: Current 1.7 changelog
Post by: Tosch110 on December 11, 2015, 07:21:56 am
With the changes of the block algorithm we expect 1440 Blocks a day when I understood that right. That means the maximum leasing range decreases from ~5 weeks to ~2,5.

Can we think about increasing the maximum leasing range?
Title: Re: Current 1.7 changelog
Post by: lurker10 on December 11, 2015, 08:02:30 am
With the changes of the block algorithm we expect 1440 Blocks a day when I understood that right. That means the maximum leasing range decreases from ~5 weeks to ~2,5.

Can we think about increasing the maximum leasing range?

+1440

5 weeks is on the borderline of convenience. Forgers will forget or won't bother to renew leases more often if the range is halved weakening security of the blockchain.
Title: Re: Current 1.7 changelog
Post by: Jean-Luc on December 11, 2015, 05:12:17 pm
We will increase the maximum leasing period to 65535 in 1.7, this will be 6.5 weeks.
Title: Re: Current 1.7 changelog
Post by: lurker10 on December 11, 2015, 06:06:20 pm
We will increase the maximum leasing period to 65535 in 1.7, this will be 6.5 weeks.

thank you!
Title: Re: Current 1.7 changelog
Post by: Tosch110 on December 12, 2015, 03:18:02 pm
We will increase the maximum leasing period to 65535 in 1.7, this will be 6.5 weeks.

thank you!

yep, great thing! thanks!
Title: Re: Current 1.7 changelog
Post by: VanBreuk on December 12, 2015, 04:07:49 pm
Account properties
Those are name / value pairs that can be set on any account (except Genesis), by either the account owner, or by another account. Names are limited to 32 characters, and values to 160 characters. Names are unique per account and per setter account, but not globally unique. Account properties cannot be transferred between accounts. The setter of an account property can edit it by replacing its value with another. Either the setter, or the recipient (if different) of an account property can delete it. There is no limit on the number of properties an account can have. Fee for setting account property is 1 NXT for value up to 32 chars, with additional 1 NXT fee for every 32 chars after that.

What are the use cases for those properties? What was the idea behind them? How can services use them? How are they useful for the regular user with the standard client?

I am also looking into these.

Can anyone shed some light on this?

From this thread (https://nxtforum.org/index.php?topic=8891.0),

I imagine a single setAccountProperty API which takes a name and a value, similar to aliases but account specific, name unique within the account only, update the value if already existing or delete the name/value pair if value empty. And getAccountProperties which returns all of them for a given account.

Any third party service could add to accounts custom key/value pairs to store relevant info for their service in the Nxt blockchain, with no need of scattering messages around. To use an example in what I know, MGW could for instance work storing deposit addresses as an Account Property, "BTCdeposit":"msigaddr".

When third party developers can assign pairs to existing user accounts (or create new accounts with the value pairs included) and then query those values, this gives the API a good extra of potential usability.
elective-stereophonic
elective-stereophonic
assembly
assembly