elective-stereophonic
elective-stereophonic
New fees and size limits in 1.7
singapore
Please login or register.

Login with username, password and session length
Advanced search  

News:

Latest Stable Nxt Client: Nxt 1.12.1 Upgrade before block 2870000 is mandatory!

Pages: 1 2 3 ... 6 [All]

Author Topic: New fees and size limits in 1.7  (Read 21442 times)

Jean-Luc

  • Core Dev
  • Hero Member
  • *****
  • Karma: +816/-81
  • Offline Offline
  • Posts: 1610
    • View Profile
New fees and size limits in 1.7
« on: November 11, 2015, 05:15:57 pm »

I would like to bring to everyone's attention the important changes regarding the new minimum fees and size limits planned to take effect at the 1.7 hardfork, as they will very likely affect all clients that have hardcoded minimum fees or use permanent messages. See the current 1.7 changelog for details: https://nxtforum.org/general-discussion/current-1-7-changelog

There are several reasons for these changes. Most importantly, with the improved support for prunable data introduced in 1.6, there is no excuse to let users continue using permanent instead of prunable messages. To encourage a switch to prunable, the fees for permanent blockchain space must be increased, and the maximum message size limit reduced. We also want to make fees proportionate to the actual blockchain space consumed, charging a fixed fee per each 32 bytes, and to make that consistent for all transaction types which store data in the blockchain. For a few transaction types (aliases, account info, DGS goods) it still makes sense to allow a higher data size limit, for those we suggest keeping the current 1000 bytes limit, but setting the per byte fee to twice that for messages, to prevent using such transactions instead of messages. For some high fee transaction types, we have also added throttling and/or back fees spreading, to prevent spam or abuse by the forgers.

It is good to have a free permanent message allowance for every transaction, and we have tentatively set this to 32 bytes. There is a dependency between the various fees however, because if the cost per 32 bytes after the free allowance is too high, this would create a loophole for abuse, with users splitting messages into multiple transactions to fit within the free limit. To prevent such abuse, the base transaction fee (currently 1 NXT) must be roughly the same as the cost per 32 bytes.

We plan to create a poll to allow NXT holders to vote on what the base fee should be, with possible values of 1, 2, 3 or 4 NXT. Then the fees for permanent blockchain storage space (cost per 32 bytes), and all other fees, will be determined relative to this base fee, e.g. if it ends up being set to 2 NXT instead of the current 1 NXT default, all fee values proposed in the 1.7 changelog will be scaled by 2x.
Logged
GPG key fingerprint: 263A 9EB0 29CF C77A 3D06  FD13 811D 6940 E1E4 240C
NXT-X4LF-9A4G-WN9Z-2R322

Sebastien256

  • Hero Member
  • *****
  • Karma: +169/-24
  • Offline Offline
  • Posts: 2823
  • ^LOOK UP^ = Nxt community!
    • View Profile
Re: New fees and size limits in 1.7
« Reply #1 on: November 11, 2015, 05:29:35 pm »

Hi Jl,

it would be good to put all your posts related to core devs in a specific section of the forum, otherwise posts in genral discussion will get burried and many people won't see them. This is obiously not the intented goal.
Seb
Logged
Please drop your ideas concerning Nxt and/or NRS in this topic -> List of feature request for Nxt and/or NRS (with the full list in OP).

cc001

  • Hero Member
  • *****
  • Karma: +68/-4
  • Offline Offline
  • Posts: 829
    • View Profile
Re: New fees and size limits in 1.7
« Reply #2 on: November 11, 2015, 05:43:46 pm »

We plan to create a poll to allow NXT holders to vote on what the base fee should be, with possible values of 1, 2, 3 or 4 NXT

Will it be possible to adapt this base fee to current market situation/NXT-price in the future? Could it be done automatically, based on some formula? Will it be also possible to change this base fee to fractions of one, for example 0.5 NXT or 0.01 NXT? IfWhen Nxt goes to the moon, 1 NXT fee might be too much.
Logged
cc001 Personal - NXT-8RXS-2SSK-RNF2-HSNL8
NxtReporting.com - The Nxt Asset Exchange Portfolio Manager with Profitability Tracking - Donations are greatly appreciated on NXT-5W4G-GAR6-JHJP-H8ZTW

bcdev

  • Hero Member
  • *****
  • Karma: +162/-22
  • Offline Offline
  • Posts: 666
    • View Profile
Re: New fees and size limits in 1.7
« Reply #3 on: November 11, 2015, 06:00:12 pm »

We plan to create a poll to allow NXT holders to vote on what the base fee should be, with possible values of 1, 2, 3 or 4 NXT
Why < 1 aren't in the range of possible values? My suggestion is to add 0.25, 0.5 and 0.75.

And please make the vote multiple choice. Single choice votes force you to vote for the lesser evil.
Logged

Seccour

  • Sr. Member
  • ****
  • Karma: +68/-15
  • Offline Offline
  • Posts: 380
    • View Profile
Re: New fees and size limits in 1.7
« Reply #4 on: November 11, 2015, 06:04:46 pm »

So the fee will be higher for permanent message only ?
Logged
SecFund : 9125535795764729261 (Asset ID)

sigwo

  • Full Member
  • ***
  • Karma: +14/-0
  • Offline Offline
  • Posts: 128
  • Founder, Darcrus
    • View Profile
    • Sigwo Technologies
Re: New fees and size limits in 1.7
« Reply #5 on: November 11, 2015, 06:08:29 pm »

Hi Jl,

it would be good to put all your posts related to core devs in a specific section of the forum, otherwise posts in genral discussion will get burried and many people won't see them. This is obiously not the intented goal.
Seb

I agree. Special section for Nxt special information.
Logged

Jean-Luc

  • Core Dev
  • Hero Member
  • *****
  • Karma: +816/-81
  • Offline Offline
  • Posts: 1610
    • View Profile
Re: New fees and size limits in 1.7
« Reply #6 on: November 11, 2015, 06:23:16 pm »

If the base fee stays at 1 NXT, fees will be same as now for regular payments, AE orders, and similar transactions as long as they don't include a message attachment. Fees will be calculated based on size for permanent message attachments, and for any transaction that includes some data fields that take blockchain space and could be abused instead of messages if allowed to be cheaper. This is why I am speaking of cost per byte, and have specified how this can be calculated for each of the affected transaction types.

What I have in mind is not a poll with several options, but a single option with allowed values range 1 to 4. Then the result should be the integer closer to the average of the votes, e.g if average is 1.57, round it up to 2.

We had a fee of 1 NXT when the price was 10 times higher, the market already took care of decreasing the fee. Why should we want to set it even lower? If the price goes up significantly, we will do another poll and adjust the fee. For now such adjustments will have to be done manually, and as each fee change requires a hard fork, are not practical to do more often than every 6 months, which is about how often we do hard forks now.
Logged
GPG key fingerprint: 263A 9EB0 29CF C77A 3D06  FD13 811D 6940 E1E4 240C
NXT-X4LF-9A4G-WN9Z-2R322

crimi

  • Hero Member
  • *****
  • Karma: +122/-11
  • Offline Offline
  • Posts: 863
    • View Profile
Re: New fees and size limits in 1.7
« Reply #7 on: November 11, 2015, 06:37:54 pm »

i was excited to see this thread, now im not  :o give me lower than 1 nxt please
Logged

bcdev

  • Hero Member
  • *****
  • Karma: +162/-22
  • Offline Offline
  • Posts: 666
    • View Profile
Re: New fees and size limits in 1.7
« Reply #8 on: November 11, 2015, 06:41:01 pm »

What I have in mind is not a poll with several options, but a single option with allowed values range 1 to 4. Then the result should be the integer closer to the average of the votes, e.g if average is 1.57, round it up to 2.
This method is biased towards middle options [2,3]. Also, if you want to vote for 2 and 1 is winning, the best option is to vote 4. Bad method IMO.
Logged

Jean-Luc

  • Core Dev
  • Hero Member
  • *****
  • Karma: +816/-81
  • Offline Offline
  • Posts: 1610
    • View Profile
Re: New fees and size limits in 1.7
« Reply #9 on: November 11, 2015, 06:51:27 pm »

So what do you suggest, 4 options with up to two choices allowed, and the option with most votes wins? If for some reason then users divide into two extreme groups, some favoring highest fee possible (forgers), some the lowest (frequent traders), one of the extremes will be chosen instead of the average which would have been more fair. Not sure what is the best way.
Logged
GPG key fingerprint: 263A 9EB0 29CF C77A 3D06  FD13 811D 6940 E1E4 240C
NXT-X4LF-9A4G-WN9Z-2R322

Sebastien256

  • Hero Member
  • *****
  • Karma: +169/-24
  • Offline Offline
  • Posts: 2823
  • ^LOOK UP^ = Nxt community!
    • View Profile
Re: New fees and size limits in 1.7
« Reply #10 on: November 11, 2015, 06:54:17 pm »

So what do you suggest, 4 options with up to two choices allowed, and the option with most votes wins? If for some reason then users divide into two extreme groups, some favoring highest fee possible (forgers), some the lowest (frequent traders), one of the extremes will be chosen instead of the average which would have been more fair. Not sure what is the best way.

I would do up to three polls, one between 1 nxt or 2 nxt. Then one between 2 and 3, and the last between 3 and 4.
If for any poll the lowest value win, you don't need to do the next poll.

That is the only fair method i believe.
« Last Edit: November 11, 2015, 06:59:41 pm by Sebastien256 »
Logged
Please drop your ideas concerning Nxt and/or NRS in this topic -> List of feature request for Nxt and/or NRS (with the full list in OP).

Jean-Luc

  • Core Dev
  • Hero Member
  • *****
  • Karma: +816/-81
  • Offline Offline
  • Posts: 1610
    • View Profile
Re: New fees and size limits in 1.7
« Reply #11 on: November 11, 2015, 06:58:11 pm »

i was excited to see this thread, now im not  :o give me lower than 1 nxt please
To give you lower than 1 NXT, something must have changed in our architecture to make it possible. Has it? Not with permanent messages, but the change was to add prunable data, done in 1.5, with a lot of improvements in 1.6. And for prunable, you do have the low fee of 0.1 NXT for 1024 bytes. Start using those instead.
Logged
GPG key fingerprint: 263A 9EB0 29CF C77A 3D06  FD13 811D 6940 E1E4 240C
NXT-X4LF-9A4G-WN9Z-2R322

XIII

  • Jr. Member
  • **
  • Karma: +26/-1
  • Offline Offline
  • Posts: 60
    • View Profile
    • NXTFLOW
Re: New fees and size limits in 1.7
« Reply #12 on: November 11, 2015, 07:02:40 pm »

We plan to create a poll to allow NXT holders to vote on what the base fee should be, with possible values of 1, 2, 3 or 4 NXT
Why < 1 aren't in the range of possible values? My suggestion is to add 0.25, 0.5 and 0.75.

And please make the vote multiple choice. Single choice votes force you to vote for the lesser evil.

Yes, market has already decreased fees. But, we need to make NXT more popular. In fact this doesn't matter what we get 1 USD cent or 3 USD cents per month with this forging. So, here we have psychological problem of using 1 NXT for basical fee. It would be perfect to decrease fees at least down to 0,5 NXT - cause this will be very strong marketing step for NXT.

I may be wrong for sure, let people judge, so we need to include in poll as was pointed - 0.1, 0.25, 0.5, 0.75 NXT 
Logged
NXTFLOW - first physical embassy of NXT
-----------------------------
Donate: NXT-SVZE-S87X-EAUU-5FZMH

kushti

  • Sr. Member
  • ****
  • Karma: +184/-5
  • Offline Offline
  • Posts: 384
  • Nxt Core & Apps Dev
    • View Profile
Re: New fees and size limits in 1.7
« Reply #13 on: November 11, 2015, 07:18:05 pm »

Low fees are killing forging economy and so just dangerous
Logged
for donations / messages: NXT-PKXM-WH25-UXXG-CJAVD (alias: kushti)

Sebastien256

  • Hero Member
  • *****
  • Karma: +169/-24
  • Offline Offline
  • Posts: 2823
  • ^LOOK UP^ = Nxt community!
    • View Profile
Re: New fees and size limits in 1.7
« Reply #14 on: November 11, 2015, 07:25:11 pm »

Guys, in any case multiple choice with average is bad as it could be manipulate (if weight by nxt holding), because the vote are count at the finish height afaik.
Logged
Please drop your ideas concerning Nxt and/or NRS in this topic -> List of feature request for Nxt and/or NRS (with the full list in OP).

Jean-Luc

  • Core Dev
  • Hero Member
  • *****
  • Karma: +816/-81
  • Offline Offline
  • Posts: 1610
    • View Profile
Re: New fees and size limits in 1.7
« Reply #15 on: November 11, 2015, 07:31:17 pm »

I like the idea about a series of polls with two choices each. We do need to shorten the voting period for each poll though, at most a week each, then it may still take up to three weeks total.
Logged
GPG key fingerprint: 263A 9EB0 29CF C77A 3D06  FD13 811D 6940 E1E4 240C
NXT-X4LF-9A4G-WN9Z-2R322

Sebastien256

  • Hero Member
  • *****
  • Karma: +169/-24
  • Offline Offline
  • Posts: 2823
  • ^LOOK UP^ = Nxt community!
    • View Profile
Re: New fees and size limits in 1.7
« Reply #16 on: November 11, 2015, 07:36:08 pm »

I like the idea about a series of polls with two choices each. We do need to shorten the voting period for each poll though, at most a week each, then it may still take up to three weeks total.

This forum need a place to post info about official Nxt poll. Note that it would be a bad idea to burry this kind of announcement in a general topic ;) ok ?
Logged
Please drop your ideas concerning Nxt and/or NRS in this topic -> List of feature request for Nxt and/or NRS (with the full list in OP).

Damelon

  • Hero Member
  • *****
  • Karma: +792/-54
  • Offline Offline
  • Posts: 2314
    • View Profile
    • Nxt Inside
Re: New fees and size limits in 1.7
« Reply #17 on: November 11, 2015, 07:36:33 pm »

We plan to create a poll to allow NXT holders to vote on what the base fee should be, with possible values of 1, 2, 3 or 4 NXT

Will it be possible to adapt this base fee to current market situation/NXT-price in the future? Could it be done automatically, based on some formula? Will it be also possible to change this base fee to fractions of one, for example 0.5 NXT or 0.01 NXT? IfWhen Nxt goes to the moon, 1 NXT fee might be too much.

I think we may have a communication mistake happening here

This is about base fees for messages, as far as I know, not base fees for normal transactions.

Jean-Luc, can you clarify this, please?

Edit: found it!

Quote
Fees and size limit changes

Several transaction types or attachments will have new fees and size limits, to encourage users to utilize the prunable versions when available, and to make fees proportionate to actual blockchain space consumed.

Aliases:
Base fee 2 NXT, with 2 NXT additional fee for each 32 chars of name plus URI total length, after the first 32 chars. Name and URI size limits remain at 100 and 1000 chars respectively.

Messages and EncryptedMessages (non-prunable):
Maximum length reduced to 160 bytes. 1 NXT fee for each 32 bytes after the first 32 bytes. For encrypted messages, the length is measured excluding the nonce and the 16 byte AES initialization vector.

Fees and size limit for prunable messages remain unchanged.

AccountInfo:
Base fee 1 NXT, with 2 NXT additional fee for each 32 chars of name plus description total length, after the first 32 chars. Name and description size limits remain at 100 and 1000 chars. AccountInfo transactions throttled at one per block.

Polls:
Base fee 10 NXT for polls with up to 20 options, and total size of poll name plus poll description plus total option length not exceeding 320 chars. For each option above 20, an additional fee of 1 NXT, and for each 32 chars after 320, an additional fee of 1 NXT. Poll creation throttled to one per block. Name, description, and option length limits remain at 100, 1000, and 100 chars respectively.

DGS Listing:
Base fee 2 NXT, with 2 NXT additional fee for each 32 chars of name plus description total length, after the first 32 chars. Name and description size limits remain at 100 and 1000 max. DGS Listing throttled at one per block.

DGS Delivery:
Base fee 1 NXT, with 2 NXT additional fee for each 32 bytes of encrypted goods data after the first 32 bytes, nonce and AES initialization bytes excluded. Encrypted goods data size limit remains 1000 bytes.
« Last Edit: November 11, 2015, 07:54:53 pm by Damelon »
Logged
Member of the Nxt Foundation | Donations: NXT-D6K7-MLY6-98FM-FLL5T
Join Nxt Slack! https://nxtchat.herokuapp.com/
Founder of Blockchain Workspace | Personal Site & Blog

Damelon

  • Hero Member
  • *****
  • Karma: +792/-54
  • Offline Offline
  • Posts: 2314
    • View Profile
    • Nxt Inside
Re: New fees and size limits in 1.7
« Reply #18 on: November 11, 2015, 07:38:04 pm »

I like the idea about a series of polls with two choices each. We do need to shorten the voting period for each poll though, at most a week each, then it may still take up to three weeks total.

This forum need a place to post info about official Nxt poll. Note that it would be a bad idea to burry this kind of announcement in a general topic ok?

VanBreuk and I are discussing reshuffling the whole forums setup at the moment to accommodate all Core Development related things into one well organised whole :)
The plan is to change it before end of week. Don't be surprised when the forums look different :)
Logged
Member of the Nxt Foundation | Donations: NXT-D6K7-MLY6-98FM-FLL5T
Join Nxt Slack! https://nxtchat.herokuapp.com/
Founder of Blockchain Workspace | Personal Site & Blog

XIII

  • Jr. Member
  • **
  • Karma: +26/-1
  • Offline Offline
  • Posts: 60
    • View Profile
    • NXTFLOW
Re: New fees and size limits in 1.7
« Reply #19 on: November 11, 2015, 07:55:04 pm »

Low fees are killing forging economy and so just dangerous

It would not kill forging, cause it is already almost dead. But it could force people messedging, spending and trading more, so it will force NXT at all, bring more users to NXT, and then force forging to be alive.

It is better to get 0.5 nxt from 1000 users, than 2 NXT from 100 users. 
Logged
NXTFLOW - first physical embassy of NXT
-----------------------------
Donate: NXT-SVZE-S87X-EAUU-5FZMH

Jean-Luc

  • Core Dev
  • Hero Member
  • *****
  • Karma: +816/-81
  • Offline Offline
  • Posts: 1610
    • View Profile
Re: New fees and size limits in 1.7
« Reply #20 on: November 11, 2015, 07:58:51 pm »

It is about fees for messages. But the catch is, because we want to have a free allowance of 32 bytes per message, to be able to attach e.g. order or account number to a payment without paying extra fee, if the fee for each 32 additional bytes is higher than the base transaction fee, one could cheat and split a long message into multiple 32 byte pieces, fitting each in the free allowance limit. For example, if fee per 32 bytes is 4 NXT, instead of paying 5 NXT for a transaction with 64 bytes message, the user will split it into two transactions with 32 bytes messages each and pay only 2 NXT, and this is worse for the blockchain than fitting it into a single transaction. So the base transaction fee must also scale up if the fee per byte is higher.

We could remove the free allowance instead, but since prunable messages necessarily leave behind a 32 bytes permanent hash, charging the user for a permanent message of less than 32 bytes doesn't make sense as moving to prunable will not help for such short messages, it adds more overhead.
Logged
GPG key fingerprint: 263A 9EB0 29CF C77A 3D06  FD13 811D 6940 E1E4 240C
NXT-X4LF-9A4G-WN9Z-2R322

cc001

  • Hero Member
  • *****
  • Karma: +68/-4
  • Offline Offline
  • Posts: 829
    • View Profile
Re: New fees and size limits in 1.7
« Reply #21 on: November 11, 2015, 08:27:14 pm »

We had a fee of 1 NXT when the price was 10 times higher, the market already took care of decreasing the fee. Why should we want to set it even lower? If the price goes up significantly, we will do another poll and adjust the fee. For now such adjustments will have to be done manually, and as each fee change requires a hard fork, are not practical to do more often than every 6 months, which is about how often we do hard forks now.

I totally agree with that. My only question is: Is the system able to use in the future (if needed) not only natural numbers of NXT as fee, but also fractions of a NXT as fees?
Logged
cc001 Personal - NXT-8RXS-2SSK-RNF2-HSNL8
NxtReporting.com - The Nxt Asset Exchange Portfolio Manager with Profitability Tracking - Donations are greatly appreciated on NXT-5W4G-GAR6-JHJP-H8ZTW

Jean-Luc

  • Core Dev
  • Hero Member
  • *****
  • Karma: +816/-81
  • Offline Offline
  • Posts: 1610
    • View Profile
Re: New fees and size limits in 1.7
« Reply #22 on: November 11, 2015, 08:30:05 pm »

Yes, fractional fees are fully supported, everything is in NQT internally. Although keeping it whole numbers may be easier for clients, and for end users.
Logged
GPG key fingerprint: 263A 9EB0 29CF C77A 3D06  FD13 811D 6940 E1E4 240C
NXT-X4LF-9A4G-WN9Z-2R322

NxtSwe

  • Hero Member
  • *****
  • Karma: +124/-9
  • Offline Offline
  • Posts: 657
    • View Profile
Re: New fees and size limits in 1.7
« Reply #23 on: November 11, 2015, 11:02:41 pm »

Yes, fractional fees are fully supported, everything is in NQT internally. Although keeping it whole numbers may be easier for clients, and for end users.
Never heard of milliNXT's or megaNQT's?  ;D
Logged
Check out the NxtLib, the .NET Framework API for the Nxt platform.

Tosch110

  • Hero Member
  • *****
  • Karma: +211/-18
  • Offline Offline
  • Posts: 2365
    • View Profile
Re: New fees and size limits in 1.7
« Reply #24 on: November 11, 2015, 11:56:33 pm »

Thank you Jean-Luc for your explanations! Very helpful.

I hope that the fee stays as 1 NXT so maybe integration can be more seemless and old feeNQT = 100000000 would still be accepted? ;)

crimi

  • Hero Member
  • *****
  • Karma: +122/-11
  • Offline Offline
  • Posts: 863
    • View Profile
Re: New fees and size limits in 1.7
« Reply #25 on: November 12, 2015, 11:37:18 am »



dont care about fiat value.
Logged

NxtSwe

  • Hero Member
  • *****
  • Karma: +124/-9
  • Offline Offline
  • Posts: 657
    • View Profile
Re: New fees and size limits in 1.7
« Reply #26 on: November 12, 2015, 03:09:19 pm »



dont care about fiat value.

To be fair, spending 1 NXT consumes the same amount of harddrive space and network bits as sending 1 million nxt.
It cost the same for me to take the flight with 1 EUR in my pocket as 1 Million EUR in my pocket ... well, with the exception of all the questions I would get from the security officers that is. But that's another matter.
Logged
Check out the NxtLib, the .NET Framework API for the Nxt platform.

crimi

  • Hero Member
  • *****
  • Karma: +122/-11
  • Offline Offline
  • Posts: 863
    • View Profile
Re: New fees and size limits in 1.7
« Reply #27 on: November 12, 2015, 03:26:47 pm »



dont care about fiat value.

To be fair, spending 1 NXT consumes the same amount of harddrive space and network bits as sending 1 million nxt.
It cost the same for me to take the flight with 1 EUR in my pocket as 1 Million EUR in my pocket ... well, with the exception of all the questions I would get from the security officers that is. But that's another matter.

Trust me if you want to send 1 btc and you have to pay 1 btc fee, bitcoin would be a dead drop today. This is impossible to market.
Logged

bcdev

  • Hero Member
  • *****
  • Karma: +162/-22
  • Offline Offline
  • Posts: 666
    • View Profile
Re: New fees and size limits in 1.7
« Reply #28 on: November 12, 2015, 03:33:43 pm »

To be fair, spending 1 NXT consumes the same amount of harddrive space and network bits as sending 1 million nxt.
It cost the same for me to take the flight with 1 EUR in my pocket as 1 Million EUR in my pocket ... well, with the exception of all the questions I would get from the security officers that is. But that's another matter.
You can't fly with more than $10k in US. Police can arrest you if you attempt that.
This includes any foreign currency. I've heard a rumor of a Canadian man who had 999 USD worth of CAD with him using the exchange rate of the day before. Sadly, the CAD/USD rose that day. Needless to say, he was arrested.
Logged

NxtSwe

  • Hero Member
  • *****
  • Karma: +124/-9
  • Offline Offline
  • Posts: 657
    • View Profile
Re: New fees and size limits in 1.7
« Reply #29 on: November 12, 2015, 03:42:45 pm »



dont care about fiat value.

To be fair, spending 1 NXT consumes the same amount of harddrive space and network bits as sending 1 million nxt.
It cost the same for me to take the flight with 1 EUR in my pocket as 1 Million EUR in my pocket ... well, with the exception of all the questions I would get from the security officers that is. But that's another matter.

Trust me if you want to send 1 btc and you have to pay 1 btc fee, bitcoin would be a dead drop today. This is impossible to market.

Are you saying that a $330 fee is too much if you want to send $330?
If so, we are in agreement.
Logged
Check out the NxtLib, the .NET Framework API for the Nxt platform.

lurker10

  • Hero Member
  • *****
  • Karma: +168/-33
  • Offline Offline
  • Posts: 1334
    • View Profile
Re: New fees and size limits in 1.7
« Reply #30 on: November 12, 2015, 03:44:49 pm »

To be fair, spending 1 NXT consumes the same amount of harddrive space and network bits as sending 1 million nxt.
It cost the same for me to take the flight with 1 EUR in my pocket as 1 Million EUR in my pocket ... well, with the exception of all the questions I would get from the security officers that is. But that's another matter.
You can't fly with more than $10k in US. Police can arrest you if you attempt that.
This includes any foreign currency. I've heard a rumor of a Canadian man who had 999 USD worth of CAD with him using the exchange rate of the day before. Sadly, the CAD/USD rose that day. Needless to say, he was arrested.

Land of the Free.
Logged
Run a node - win a prize! "Lucky node" project jar: NXT-8F28-EDVE-LPPX-HY4E7

dude

  • Full Member
  • ***
  • Karma: +44/-5
  • Offline Offline
  • Posts: 207
    • View Profile
Re: New fees and size limits in 1.7
« Reply #31 on: November 12, 2015, 03:45:39 pm »

Trust me if you want to send 1 btc and you have to pay 1 btc fee, bitcoin would be a dead drop today. This is impossible to market.

I don't see how having the fee at 0.5 for example could help this. The answer is, don't market it like that (you can express fees in fiat) or not at all, market useful stuff.

edit: Also, if I could get 150 BTC for $1, I wouldn't care.
« Last Edit: November 12, 2015, 03:52:11 pm by dude »
Logged

Damelon

  • Hero Member
  • *****
  • Karma: +792/-54
  • Offline Offline
  • Posts: 2314
    • View Profile
    • Nxt Inside
Re: New fees and size limits in 1.7
« Reply #32 on: November 12, 2015, 04:02:31 pm »



dont care about fiat value.

To be fair, spending 1 NXT consumes the same amount of harddrive space and network bits as sending 1 million nxt.
It cost the same for me to take the flight with 1 EUR in my pocket as 1 Million EUR in my pocket ... well, with the exception of all the questions I would get from the security officers that is. But that's another matter.

Trust me if you want to send 1 btc and you have to pay 1 btc fee, bitcoin would be a dead drop today. This is impossible to market.

This example only works with numbers that are large.

Listen to CoinOutlet: https://www.youtube.com/watch?v=FN0gUIVYu2s
They actually chose NXT because they thought the TX fee is low. And today, they would be right.

We already know that once the fees would become prohibitive (probably around 5 ct, but don't pin me down on it), the base fee could and would be changed. NXT is well prepared for that.

At this moment though, the fee in fiat is not the problem.

The only thing that could be against it is that relatively the fee is high, as when you consider them within the ecosystem.

This is a delicate balance though, because as long as the platform is cheap in marketcap, it's rather easy to misuse it when we'd lower the fees at this point.

We have never really had this issue crop up when talking to anyone. Mostly they say "what? 1 ct per tx? That's reasonable". Try to sell it in terms of fiat, and you're golden. Try to sell it in terms of the internal market of the platform, then you're in trouble.
Logged
Member of the Nxt Foundation | Donations: NXT-D6K7-MLY6-98FM-FLL5T
Join Nxt Slack! https://nxtchat.herokuapp.com/
Founder of Blockchain Workspace | Personal Site & Blog

lurker10

  • Hero Member
  • *****
  • Karma: +168/-33
  • Offline Offline
  • Posts: 1334
    • View Profile
Re: New fees and size limits in 1.7
« Reply #33 on: November 12, 2015, 04:05:15 pm »

Expressing fees in fiat works best.

If you calculate electricity/hardware fees, I saw a figure it actually costs $30 to send one Bitcoin transaction. These expenses are now subsidized through mining of new coins but what do you get in a few years when Bitcoin block rewards go down? NXT doesn't have this subsidy, on the other hand it doesn't require expensive hardware or paying for electricity much. 1 NXT has worked as a reasonable fee while 1 NXT < $0.10, let it stay this way. When the price goes above $0.10, the fees may become a burden, not yet. You can't expect to be able to write to a global ledger for nothing, it's a very limited resource. Hopefully CfB's Iota can enable micro transactions for milli-fractions of a US cent :)
Logged
Run a node - win a prize! "Lucky node" project jar: NXT-8F28-EDVE-LPPX-HY4E7

Audo Kryptowitz

  • Jr. Member
  • **
  • Karma: +69/-2
  • Offline Offline
  • Posts: 94
    • View Profile
Re: New fees and size limits in 1.7
« Reply #34 on: November 12, 2015, 04:58:22 pm »


Larger fees encourage people to secure the network.

The fees won't just vanish, rather they will go to those who provide a very important service with their hard earned NXT.
Logged
An Open Source Self-organizing Ecosystem - www.digitalcatallaxy.com

Hachoir

  • Full Member
  • ***
  • Karma: +12/-13
  • Offline Offline
  • Posts: 113
    • View Profile
Re: New fees and size limits in 1.7
« Reply #35 on: November 12, 2015, 05:04:09 pm »

Yes, perhaps more people will forge when it's profitable.
Logged

Brangdon

  • Hero Member
  • *****
  • Karma: +229/-25
  • Offline Offline
  • Posts: 1389
  • Quality is addictive.
    • View Profile
Re: New fees and size limits in 1.7
« Reply #36 on: November 12, 2015, 09:13:27 pm »

Low fees are killing forging economy and so just dangerous
I don't think they are, because whatever the fees, the income from forging is insignificant at current usage levels. From what I've read in previous discussions, the current fees are high enough to discourage use in third world countries, and for certain types of transaction such as updating a bid; and this is also dangerous.

So we disagree. We can debate it some, but eventually shouldn't we resolve this disagreement with a vote? If you succeed in keeping the options you dislike off the ballot paper, then that pre-empts the vote result. If you are right, then allowing the lower fee options is harmless, because the forgers will all vote for higher fees. (And, assuming it will be vote-by-account-balance, the big forgers are the only votes that matter.)

The other argument against low fees is spam protection. I'm more sympathetic to this argument. I'd still like to see lower fees, but maybe coupled to some other anti-spam mechanism, such as coin age or PoW. That could be some future update.
Logged

petko

  • Full Member
  • ***
  • Karma: +24/-0
  • Offline Offline
  • Posts: 100
    • View Profile
    • My blog
Re: New fees and size limits in 1.7
« Reply #37 on: November 12, 2015, 10:01:16 pm »

I still don't get it why minimal fees need to be enforced as part of the consensus.

Blockchain bloat will be better handled by a more adequate limit on the block size. Limit in bytes, not number of transactions. Like in Bitcoin. (Plus the impact each transaction has on the DB size can be included in the evaluation of each transaction).
I don't mean there won't be fees, I mean they won't be part of the consensus. Each forger will decide what minimal fees to require - it's better than voting with polls. And fluctuations in NXT price won't be a problem any more.

All unique names can be substituted by a nullable reference to an alias. We can have a currency or asset without name, and identified only by ID, why not? If the creator wants recognizable name, he/she attaches an alias. Aliases will expire. This way aliases belonging to lost account are not a problem any more, and there is no need of 25000 NXT minimal fee for a currency code.
Logged

Jean-Luc

  • Core Dev
  • Hero Member
  • *****
  • Karma: +816/-81
  • Offline Offline
  • Posts: 1610
    • View Profile
Re: New fees and size limits in 1.7
« Reply #38 on: November 13, 2015, 06:41:51 am »

I still don't get it why minimal fees need to be enforced as part of the consensus.

Blockchain bloat will be better handled by a more adequate limit on the block size. Limit in bytes, not number of transactions. Like in Bitcoin. (Plus the impact each transaction has on the DB size can be included in the evaluation of each transaction).
I don't mean there won't be fees, I mean they won't be part of the consensus. Each forger will decide what minimal fees to require - it's better than voting with polls. And fluctuations in NXT price won't be a problem any more.
We have discussed this before already. At the current transaction load, block size limit is not a factor that can influence the fees, unless we reduce it more than 10 times (and we do have two separate limits, both on transaction count and total transaction bytes size). Only when we get enough transaction traffic so that a significant percentage of the blocks are full, then a market rate fee can develop, by transaction senders willing to pay more in order to get their transaction included faster. With the current situation, if forgers can decide the fee it will gravitate towards zero, and expose the blockchain to spam. And we must think about the future, every MB of permanent data stored in the blockchain now is here to stay forever (the prunable side chains planned for 2.0 are not yet a reality and not guaranteed to be possible to implement). Migration to prunable data must be forced, leaving the permanent option only for those who really need it and are willing to pay the high fees for it.

Quote
All unique names can be substituted by a nullable reference to an alias. We can have a currency or asset without name, and identified only by ID, why not? If the creator wants recognizable name, he/she attaches an alias. Aliases will expire. This way aliases belonging to lost account are not a problem any more, and there is no need of 25000 NXT minimal fee for a currency code.
Could be. This is a theoretical discussion at this point, we have a working product and are not going to rewrite it in a different way that may or may not be better.
« Last Edit: November 13, 2015, 08:32:51 am by Jean-Luc »
Logged
GPG key fingerprint: 263A 9EB0 29CF C77A 3D06  FD13 811D 6940 E1E4 240C
NXT-X4LF-9A4G-WN9Z-2R322

Jean-Luc

  • Core Dev
  • Hero Member
  • *****
  • Karma: +816/-81
  • Offline Offline
  • Posts: 1610
    • View Profile
Re: New fees and size limits in 1.7
« Reply #39 on: November 13, 2015, 07:08:45 am »

Before someone complains that those fee and size changes have caught them unprepared and have not been announced in advance, here is a history of my posts on the topic, documenting that I have warned multiple times about this planned fees increase.

I first proposed that an increase of permanent AM fees should happen in 1.5.1e, when prunable data was implemented:

The size of regular plain and encrypted messages in this release has been
restricted back to 1000 bytes, and will likely be reduced even further, before
the stable release. This will be done in order to encourage users to switch to
prunable instead of regular messages. Fees for regular message attachments
will also be increased substantially.


We didn't do it for the 1.5 stable only because we decided to give people more time, until 1.6 (which was at the time expected to be a hard fork):

Quote
The size of regular plain and encrypted messages in this release has been
restricted back to 1000 bytes, and will likely be reduced even further, before
the stable release. This will be done in order to encourage users to switch to
prunable instead of regular messages. Fees for regular message attachments
will also be increased substantially.
It will break compatibility. I don't think that 1000 bytes limit should be lowered and the regular fee should be increased.
Of course it will break compatibility, but we can't stick with a wrong decision forever. 1.5 is expected to stay on testnet for long time. Even if we don't do the fee or size changes immediately in 1.5, they will have to be done in 1.6. One reason for reusing the same parameter names for prunable and permanent messages was to make it easy to switch, you only need to add messageIsPrunable=true.

1000 bytes for 1 NXT is not a wrong decision. You will have most of the messages prunable by simply making them default in the NXT client. And you can make prunnable messages fee slightly less to speed up transition to new messages. By breaking compatibility and raising fees you will only push away NXT services.
There are currently 130,000 transactions with unencrypted messages and only 10,000 with encrypted. The client defaults to encrypted, all plain ones must have come from API users. Changing the default to prunable in the client is not enough.

It is a tradeoff, the more we delay, the more API users will need to make the switch. Long term there is no excuse for keeping those verbose json messages permanently in the blockchain. I think permanent message size should eventually be limited to around 100 bytes for no extra fee, to fit things like order numbers, bitcoin addresses, account numbers or public keys. Above that, 1 NXT per 100 bytes with a hard limit of 1000.

The fees I gave were just an example and to start a discussion about it. But whether we increase fees now or in 1.6, or sometime in between, the long term direction has to be switching to prunable. Only data that the blockhain itself needs to verify and process future transactions should be permanently stored in it. Data that are only meaningful to external services, should be prunable after they have been spread around and recorded by archival nodes / service providers.

A few months later, the impending permanent message fee increase was mentioned again:

what is the size limit for "message" attachment that wont get pruned?
The limit for non-prunable messages has not changed, it is still 1000 bytes. Messages are pruned not based on size, but only if they have been created as prunable, i.e. messageIsPrunable=true parameter has been added. For prunable, the limit is 42 kbytes.

For 1.6 we should either reduce the size limit for non-prunable, or make them more expensive, to encourage the use of prunable messages instead.

And again:

You should run a few nodes with pruning disabled, where those are kept longer or indefinitely, from which older messages can be retrieved by those interested in them.

Expect the fee for permanent messages to increase significantly in 1.6, to motivate users to switch to prunable.


And again:

For 1.6, the fees for permanent messages will actually increase a lot. This is to force migration to prunable, there is no reason for people to continue using permanent messages now that prunable are available.

And again:

Concatenated arbitrary messages full of verbose JSON are another example of the fee not being a consideration. If it were, we would be seeing messages in binary, compressed, or at least minified json. Yet there is plenty of very verbose JSON in those, taking multiple messages instead of one, because it is more convenient for the programmers, and the fee is too low to make them worry about it. And they don't care about blockchain size either, or they would have tried to reduce message size even when it takes a single message only. To make them care, we will increase the fees for permanent messages.
Logged
GPG key fingerprint: 263A 9EB0 29CF C77A 3D06  FD13 811D 6940 E1E4 240C
NXT-X4LF-9A4G-WN9Z-2R322

petko

  • Full Member
  • ***
  • Karma: +24/-0
  • Offline Offline
  • Posts: 100
    • View Profile
    • My blog
Re: New fees and size limits in 1.7
« Reply #40 on: November 13, 2015, 06:56:46 pm »

We have discussed this before already. At the current transaction load, block size limit is not a factor that can influence the fees, unless we reduce it more than 10 times (and we do have two separate limits, both on transaction count and total transaction bytes size). Only when we get enough transaction traffic so that a significant percentage of the blocks are full, then a market rate fee can develop, by transaction senders willing to pay more in order to get their transaction included faster. With the current situation, if forgers can decide the fee it will gravitate towards zero, and expose the blockchain to spam. And we must think about the future, every MB of permanent data stored in the blockchain now is here to stay forever (the prunable side chains planned for 2.0 are not yet a reality and not guaranteed to be possible to implement). Migration to prunable data must be forced, leaving the permanent option only for those who really need it and are willing to pay the high fees for it.
We discussed this but I didn't fully understand your arguments and you didn't answer the last mail. So with the current algorithm we say "the network supports blocks with size MAX_PAYLOAD_LENGTH, but we are not going to waste space for transactions with fee below certain value". OK, makes sense.
Logged

Brangdon

  • Hero Member
  • *****
  • Karma: +229/-25
  • Offline Offline
  • Posts: 1389
  • Quality is addictive.
    • View Profile
Re: New fees and size limits in 1.7
« Reply #41 on: November 14, 2015, 12:54:10 pm »

We could remove the free allowance instead, but since prunable messages necessarily leave behind a 32 bytes permanent hash, charging the user for a permanent message of less than 32 bytes doesn't make sense as moving to prunable will not help for such short messages, it adds more overhead.
Would it make sense if we also change the charging for prunable attachments? In 1.5.1e that was "0.1 NXT per 1024 bytes, with the first 1024 bytes free (the regular transaction fee depending on its type still applies)". Is that still right, or did I miss a change log entry?

I am thinking along the lines of:
  • 1 NXT for payment with no attachment.
  • 5 NXT for payment with up to 32 bytes permanent attachment.
  • 5.1 NXT for payment with up to 1024 bytes prunable attachment.
  • Add 4 NXT for each further 32 bytes of permanent data.
  • Add 0.1 NXT for each further 1024 bytes of prunable data.
Or to put it another way, either 1 NXT, or 1 NXT + 4 NXT per 32 bytes of permanent data, or 5 NXT plus 0.1 NXT per 1024 bytes of prunable data. I think with a fee structure like this, the cheapest way to send data of a given type is always the way we'd prefer them to send it, without exploitable loopholes. The key is that the fee for prunable data includes the fee for 32 bytes of permanent data. We then have three variables to tune rather than one. For example we could set the base fee to 0.5 NXT, the permanent rate to 3 NXT, and the prunable rate at 0.1.

My goal is to allow simple transactions to be cheap, so that it is cheap to bid for assets, for example. I appreciate that some use cases, such as payment processors in Africa, really need cheap attachments too. I don't know what can be done about that.

Is this too complicated?
Logged

Brangdon

  • Hero Member
  • *****
  • Karma: +229/-25
  • Offline Offline
  • Posts: 1389
  • Quality is addictive.
    • View Profile
Re: New fees and size limits in 1.7
« Reply #42 on: November 14, 2015, 12:58:02 pm »

Low fees are killing forging economy and so just dangerous

It would not kill forging, cause it is already almost dead.
We currently have 43.4% of stake forging. Isn't that roughly what it's been for the last year or so? It's not ideal, but it's not dead, either. I guess it reflects a number of whales who don't forge because they want to protect their anonymity.
Logged

LocoMB

  • Hero Member
  • *****
  • Karma: +101/-37
  • Offline Offline
  • Posts: 751
    • View Profile
Re: New fees and size limits in 1.7
« Reply #43 on: November 14, 2015, 01:37:56 pm »

Apologies for not reading the whole thread.

I would like to float the following idea:

Pegging a base fee like a message or sendMoney TX to an average basket of some real world utility, for example 50th of a surface mail letter.

Such a physical utility fee is under physical cost constraints, because the letter has to be physically handled and expedited, and that is always related to resource usage.

PoW is linked to electricity costs- this scheme would create a somehwat similar link too.
« Last Edit: November 14, 2015, 01:41:42 pm by LocoMB »
Logged
TOX
90E54E5B5213290EE616D425CADC473038CFABFA53C913271AA8559D1937DC4AF3A354A9E4E5

blackyblack1

  • Hero Member
  • *****
  • Karma: +165/-82
  • Offline Offline
  • Posts: 1764
    • View Profile
Re: New fees and size limits in 1.7
« Reply #44 on: November 14, 2015, 02:01:18 pm »

Jean-Luc please consider increasing permanent size limit to 1000 bytes. I think people would prefer to pay extra fee than splitting the long message to 160 bytes.
Logged

NXT ENTERPRISE

  • Full Member
  • ***
  • Karma: +35/-14
  • Offline Offline
  • Posts: 194
    • View Profile
    • unanimously one
Re: New fees and size limits in 1.7
« Reply #45 on: November 14, 2015, 02:05:38 pm »

If it is an issue or argument on price costs - for me its not based in reality..

A cost of 1 - 10 nxt.... nothing.

But the scam complaint is that NXT went to a few.

People have an idea they can earn BTC by mining or earning.

So they feel like there is still a piece of the pie to earn into. not so with nxt.

Although BTC mining is not real anymore... on a small to medium scale.

So for me is it not an idea at these low prices to allow substantial NXT costs to allow an earning in forging.

All the tokens are created so shouldn't forging spread the wealth?

I would like to see more of a dispersion in forging success regardless of fee costs.

Whats 5 nxt in dollar terms today and even if nxt went to 10c a nxt?

Is it a good thing for NXT that more people are actively forging or it does not matter?

My chances of a success forge if holding under 5000,000 nxt make me not even bother.

Is a broad forging good for NXT or its not a factor?
Logged

Brangdon

  • Hero Member
  • *****
  • Karma: +229/-25
  • Offline Offline
  • Posts: 1389
  • Quality is addictive.
    • View Profile
Re: New fees and size limits in 1.7
« Reply #46 on: November 14, 2015, 04:09:01 pm »

NXT ENTERPRISE, I'm not sure what you're suggesting. Forging doesn't really redistribute wealth because it goes in proportion to how much wealth you already have. And however much you have, the return on investment is going to be insignificant compared to other things you could do with your savings, even if the fees increase 10-fold. The motivation to forge is to secure the network, not fees, and I expect it will stay that way until usage increases 100-fold.
Logged

NXT ENTERPRISE

  • Full Member
  • ***
  • Karma: +35/-14
  • Offline Offline
  • Posts: 194
    • View Profile
    • unanimously one
Re: New fees and size limits in 1.7
« Reply #47 on: November 14, 2015, 04:50:07 pm »

yes thanks for that explanation.

For BTC it is easy to say - power counts.

We need a clarification of forging..

How does average joe involve himself.

Why bother etc.

I think NXT has a PR point of confusion.

We need to clarify the small points to gain momentum for new comers.

Everything has a timing and I am confident with J-L as the core.

i do think he is playing chess with many moves ahead.

The latest talk of voting everything in is silly.

You cannot vote a bunch of non-understanding to a point of success.

But as forging is an algorithm.

How can it be easily understood?

Can forging remain as it is and a collection be made?

ex.. forge as is, collect 1500 nxt and then offer it random as a jackpot.

People love to gamble, like a curse.

once and a while some forger gets a 1500 jack pot.

easily explained the benefits now of forging...



Logged

Jean-Luc

  • Core Dev
  • Hero Member
  • *****
  • Karma: +816/-81
  • Offline Offline
  • Posts: 1610
    • View Profile
Re: New fees and size limits in 1.7
« Reply #48 on: November 14, 2015, 06:41:53 pm »

Jean-Luc please consider increasing permanent size limit to 1000 bytes. I think people would prefer to pay extra fee than splitting the long message to 160 bytes.
What I want is people to migrate to prunable, instead of splitting a long message into several small, and the inconvenience of such splitting should be another reason to do that. Just high fee isn't enough though when you reach very high fees, for example if we have 5 NXT per 32 bytes a 1000 bytes message would cost 156 NXT. For such expensive transactions we should start worrying about forger's abuse, either a forger waiting for his turn and filling a block with his own messages, or setting up a reseller service to includes others' messages for cheaper. For other expensive transactions types, we counteract this by throttling and spreading back of transaction fees. But for messages, this doesn't work because messages can be attached to any transaction type. We cannot throttle all transactions, and if we spread back the fees for all transactions it really changes the economy of forging, I prefer not to do such drastic change.
Logged
GPG key fingerprint: 263A 9EB0 29CF C77A 3D06  FD13 811D 6940 E1E4 240C
NXT-X4LF-9A4G-WN9Z-2R322

NXT ENTERPRISE

  • Full Member
  • ***
  • Karma: +35/-14
  • Offline Offline
  • Posts: 194
    • View Profile
    • unanimously one
Re: New fees and size limits in 1.7
« Reply #49 on: November 14, 2015, 06:52:53 pm »

yes very concise.

But not so much in layman's terms.

What is your priority?

NOT overloading a block

or allowing financial payments for consuming?

also can you define prunable -

so a short message will show a smaller fee and a longer message will charge more?

So a user can see and intemperate their coming expenses?

Can a forger start to see what a block is worth?



Logged

Jean-Luc

  • Core Dev
  • Hero Member
  • *****
  • Karma: +816/-81
  • Offline Offline
  • Posts: 1610
    • View Profile
Re: New fees and size limits in 1.7
« Reply #50 on: November 14, 2015, 06:57:04 pm »

Would it make sense if we also change the charging for prunable attachments? In 1.5.1e that was "0.1 NXT per 1024 bytes, with the first 1024 bytes free (the regular transaction fee depending on its type still applies)". Is that still right, or did I miss a change log entry?
For now fee for prunable remains unchanged, 0.1 NXT per 1024 bytes. If we do set a base fee 2x the current 1 NXT and scale up all fees twice, it would make sense to be consistent and increase this also to 0.2 NXT, but this will still be cheap enough.
Quote
Or to put it another way, either 1 NXT, or 1 NXT + 4 NXT per 32 bytes of permanent data, or 5 NXT plus 0.1 NXT per 1024 bytes of prunable data. I think with a fee structure like this, the cheapest way to send data of a given type is always the way we'd prefer them to send it, without exploitable loopholes. The key is that the fee for prunable data includes the fee for 32 bytes of permanent data. We then have three variables to tune rather than one. For example we could set the base fee to 0.5 NXT, the permanent rate to 3 NXT, and the prunable rate at 0.1.
What you are suggesting is removing the free allowance completely, and charging for those first 32 bytes, even in the case of prunable messages. I prefer to leave the free allowance, it is useful in many cases. In particular, it is better if merchants or exchanges use a small message attachment, with the order number or user account number, to indicate the purpose of each payment, rather than creating a new one-time use account for each payment, such account creation is more expensive for us. If we start charging extra even for those first 32 bytes, we completely discourage such use.
Quote
My goal is to allow simple transactions to be cheap, so that it is cheap to bid for assets, for example. I appreciate that some use cases, such as payment processors in Africa, really need cheap attachments too. I don't know what can be done about that.
About the needs of such payment processors, there is just orders of magnitude difference between the fees they can afford and the Western standards, there is just no way to have a fee that is cheap enough for them, yet high enough both to prevent spam and to ensure some economic rewards for forgers, in the rest of the world. I think the conclusion was that we need the prunable side chains, before we can think of special low fees for those countries, and even then it is not clear how they would be protected from potential spam attacks coming from elsewhere.

Logged
GPG key fingerprint: 263A 9EB0 29CF C77A 3D06  FD13 811D 6940 E1E4 240C
NXT-X4LF-9A4G-WN9Z-2R322

Brangdon

  • Hero Member
  • *****
  • Karma: +229/-25
  • Offline Offline
  • Posts: 1389
  • Quality is addictive.
    • View Profile
Re: New fees and size limits in 1.7
« Reply #51 on: November 14, 2015, 08:40:39 pm »

What you are suggesting is removing the free allowance completely, and charging for those first 32 bytes, even in the case of prunable messages. I prefer to leave the free allowance, it is useful in many cases. In particular, it is better if merchants or exchanges use a small message attachment, with the order number or user account number, to indicate the purpose of each payment, rather than creating a new one-time use account for each payment, such account creation is more expensive for us. If we start charging extra even for those first 32 bytes, we completely discourage such use.
OK, thanks.
Logged

chanc3r

  • Hero Member
  • *****
  • Karma: +124/-50
  • Offline Offline
  • Posts: 1019
  • NXTInspect
    • View Profile
Re: New fees and size limits in 1.7
« Reply #52 on: November 14, 2015, 09:19:36 pm »

Quote

Messages and EncryptedMessages (non-prunable):
Maximum length reduced to 160 bytes. 1 NXT fee for each 32 bytes after the first 32 bytes. For encrypted messages, the length is measured excluding the nonce and the 16 byte AES initialization vector.

I'm posting this here and in the other threads so its on record...
A Maximum permanent message length of 160 will cause some MGW transactions to fail unless this MGW is changed or this length is increased, still figuring out the minimum needed to sustain MGW unchanged but 160 is definitely too low. This will also increase MGW transaction costs depending on the base fee - for those of you voting...
Logged
NXT: 29996814460165 (NXT-JTA7-B2QR-8BFC-2V222)
@imrimr @NXTinspect

Jean-Luc

  • Core Dev
  • Hero Member
  • *****
  • Karma: +816/-81
  • Offline Offline
  • Posts: 1610
    • View Profile
Re: New fees and size limits in 1.7
« Reply #53 on: November 14, 2015, 09:22:45 pm »

The MGW must switch to prunable. This is what prunable messages are for.
Logged
GPG key fingerprint: 263A 9EB0 29CF C77A 3D06  FD13 811D 6940 E1E4 240C
NXT-X4LF-9A4G-WN9Z-2R322

chanc3r

  • Hero Member
  • *****
  • Karma: +124/-50
  • Offline Offline
  • Posts: 1019
  • NXTInspect
    • View Profile
Re: New fees and size limits in 1.7
« Reply #54 on: November 14, 2015, 09:31:43 pm »

The MGW must switch to prunable. This is what prunable messages are for.
Fine BUT this means changing all the asset transfer code across all of the different MGW builds (they are not all on the same version as I understand it) and we are not talking about going back to 1k per AM but more likely to something like 180-200 chars so save them a huge amount of work.

MGW uses the blockchain as a ledger and does a full rescan when it boots so i needs all messages.
I know this can be done with nodes running the prunable service but its still significant coding work and testing for the MGW teams to switch MGW to create prunable messages as part of the asset transfers.
Logged
NXT: 29996814460165 (NXT-JTA7-B2QR-8BFC-2V222)
@imrimr @NXTinspect

Jean-Luc

  • Core Dev
  • Hero Member
  • *****
  • Karma: +816/-81
  • Offline Offline
  • Posts: 1610
    • View Profile
Re: New fees and size limits in 1.7
« Reply #55 on: November 14, 2015, 09:37:50 pm »

To use create a prunable instead of permanent message, the only change is to add messageIsPrunable=true parameter to the asset transfer (or any other) create transaction call.

It is intentional that in the transaction JSON, both permanent and prunable messages appear as the same "message" field (and this is one reason you can't have both at the same time), this is to make the switch to prunable easier as parsing the JSON attachment does not need to change.
Logged
GPG key fingerprint: 263A 9EB0 29CF C77A 3D06  FD13 811D 6940 E1E4 240C
NXT-X4LF-9A4G-WN9Z-2R322

chanc3r

  • Hero Member
  • *****
  • Karma: +124/-50
  • Offline Offline
  • Posts: 1019
  • NXTInspect
    • View Profile
Re: New fees and size limits in 1.7
« Reply #56 on: November 14, 2015, 09:46:12 pm »

I completely understand the change how this is implemented in the JSON and the lengths you have gone to ease the decoding of messages are clear to me in the API from using it myself.

What is needed to implement it is exactly what you have outlined but this means MGW itself going through a development lifecycle that currently isn't planned I suspect and it may be this call isn't isolated in one place but in several.

Forgive me I don't understand the material difference between every node storing 160byte messages forever and storing 180-200 byte messages forever. MOST MGW messages are less than 160 - its just *a few* are not.

Core-devs change the current proposal by a few bytes and another community team is saved a lot of work.

If the deadline for the switch is December - I suspect this will be challenging for the Supernet team because in this case based on what I saw in the MGW code a while ago this is a more difficult change than accommodating the 1.6.2 change
Logged
NXT: 29996814460165 (NXT-JTA7-B2QR-8BFC-2V222)
@imrimr @NXTinspect

Jean-Luc

  • Core Dev
  • Hero Member
  • *****
  • Karma: +816/-81
  • Offline Offline
  • Posts: 1610
    • View Profile
Re: New fees and size limits in 1.7
« Reply #57 on: November 14, 2015, 09:58:02 pm »

The deadline is not Dec 1st, the hard fork will take place probably early January. Furthermore, if it is a matter of a few more weeks only, we can encode two different hard fork blocks in the same release, so that shuffling and other new features for example get enabled on Jan 5th, the fee increase and size limit changes take effect on Jan 15th.

It is not the difference between 160 and 200 bytes, it is the difference between 200 and 32 bytes. I want the MGW team to make the switch to prunable, even if it means extra work, and even possibly doing a two stage hard fork as described above.

Ever since the prunable feature was released in April, I have repeatedly stated that projects like MGW should migrate to it, yet they kept ignoring this. I have no doubt that if we let them use 200 bytes permanent AMs, MGW will plan to use permanent forever, and to save themselves a few weeks of work (and grepping through the code to add a parameter, even if in many places, doesn't take weeks), they will not care if this causes blockchain bloat for everyone else using the platform. Tragedy of the commons.
Logged
GPG key fingerprint: 263A 9EB0 29CF C77A 3D06  FD13 811D 6940 E1E4 240C
NXT-X4LF-9A4G-WN9Z-2R322

Brangdon

  • Hero Member
  • *****
  • Karma: +229/-25
  • Offline Offline
  • Posts: 1389
  • Quality is addictive.
    • View Profile
Re: New fees and size limits in 1.7
« Reply #58 on: November 14, 2015, 10:43:02 pm »

Core-devs change the current proposal by a few bytes and another community team is saved a lot of work.
But you'll have to do that work eventually anyway, right? 200 bytes even at 1 NXT per 32 bytes is 7 NXT, and if you switch to prunable data it would cost only 1 NXT. However the vote goes, it will cost 1/7th as much to change. Surely SuperNet would prefer the cheaper option? Or do you not really care, because you pass the costs onto your users?

That said, I would be interested in the rationale for the limit being 160 bytes, rather than, say, 128. Is there some specific use-case it is catering to? Is it the average transaction size (without attachments) - I gather that's usually between 150 and 200 bytes?
Logged

chanc3r

  • Hero Member
  • *****
  • Karma: +124/-50
  • Offline Offline
  • Posts: 1019
  • NXTInspect
    • View Profile
Re: New fees and size limits in 1.7
« Reply #59 on: November 14, 2015, 11:06:32 pm »

The deadline is not Dec 1st, the hard fork will take place probably early January. Furthermore, if it is a matter of a few more weeks only, we can encode two different hard fork blocks in the same release, so that shuffling and other new features for example get enabled on Jan 5th, the fee increase and size limit changes take effect on Jan 15th.

J-L thanks for the suggestion, in any case I think it would make sense to bring in the new features on one fork and implement the more potentially disruptive change on another fork, will discuss timelines with the supermet team to make the change.

I understand, also, this change is needed to ensure the MGW TX remain cost effective as prunable will be cheaper than permanent anyway.
Logged
NXT: 29996814460165 (NXT-JTA7-B2QR-8BFC-2V222)
@imrimr @NXTinspect

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: New fees and size limits in 1.7
« Reply #60 on: November 14, 2015, 11:54:05 pm »

The deadline is not Dec 1st, the hard fork will take place probably early January. Furthermore, if it is a matter of a few more weeks only, we can encode two different hard fork blocks in the same release, so that shuffling and other new features for example get enabled on Jan 5th, the fee increase and size limit changes take effect on Jan 15th.

It is not the difference between 160 and 200 bytes, it is the difference between 200 and 32 bytes. I want the MGW team to make the switch to prunable, even if it means extra work, and even possibly doing a two stage hard fork as described above.

Ever since the prunable feature was released in April, I have repeatedly stated that projects like MGW should migrate to it, yet they kept ignoring this. I have no doubt that if we let them use 200 bytes permanent AMs, MGW will plan to use permanent forever, and to save themselves a few weeks of work (and grepping through the code to add a parameter, even if in many places, doesn't take weeks), they will not care if this causes blockchain bloat for everyone else using the platform. Tragedy of the commons.
MGW uses the blockchain as its permanent record of deposits and withdraws. My understanding is that using prunable transactions means it disappears after 2 weeks.

How would you suggest to correlate the transactions made if the data is pruned away?

So it is not a matter of changing API calls. By using the blockchain, the user transactions are there, regardless of what happens to the servers. MGW recalculates balances by scanning the blockchain for all transactions, you know, like a blockchain.

I am confused why you keep advocating MGW use prunable data. Where should I store the user transactions info. These are part of the deposit/withdraws so there is no chance of them being separated and lost. Once we start storing the user transaction data in any other place than the actual transaction, then it loses the blockchain aspect it has.

Any fee increase will need to be passed on to end users as MGW fees are not covering costs as it is.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: New fees and size limits in 1.7
« Reply #61 on: November 15, 2015, 12:08:14 am »

Core-devs change the current proposal by a few bytes and another community team is saved a lot of work.
But you'll have to do that work eventually anyway, right? 200 bytes even at 1 NXT per 32 bytes is 7 NXT, and if you switch to prunable data it would cost only 1 NXT. However the vote goes, it will cost 1/7th as much to change. Surely SuperNet would prefer the cheaper option? Or do you not really care, because you pass the costs onto your users?

That said, I would be interested in the rationale for the limit being 160 bytes, rather than, say, 128. Is there some specific use-case it is catering to? Is it the average transaction size (without attachments) - I gather that's usually between 150 and 200 bytes?
Prunable storage of people's deposit/withdraw info?

That will open up a lot of new edge cases. I certainly dont want to rely on public servers keeping permanent copies of all such info and there is no guarantee that there would be any such servers with a permanent record

so the data needs to be in the permanent data. I keep saying this over and over

permanent data and it is part of the asset transfers, so there is no chance of them getting lost.

I certainly dont have a month (yes, debugging MGW is very timeconsuming) to code and debug an MGW that uses prunable messages (using remote servers, setting up some internal servers as archive nodes, testing remote retrieval, etc). Even if I had the time, this prunable design would degrade MGW reliability by making it a LOT more complicated than now. Plus it would depend on public servers for proper operations

The only practical solution i see if txfees go up is to increase the MGW fee, that is not a problem for me to do. I will just charge 10NXT txfee for each MGW tx, which would cover the worst case costs and a bit extra

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

xiahui135

  • Full Member
  • ***
  • Karma: +15/-11
  • Offline Offline
  • Posts: 150
    • View Profile
Re: New fees and size limits in 1.7
« Reply #62 on: November 15, 2015, 03:52:50 am »

why not make the base fee changeable in the block-chain?
If nxt price go more than 10x higher, the fee is too high. We should prepare for the high price, not to limit the price to go high.

Before, the price is main determined by speculation when people just buy in a centralized exchange. Specutors do not care much about the transation fee, because it will not affect them much.
But we aim to make the price determined by application in future, if nxt will to success. Given this, we need to encourage usage of the our block chain, and lower fee make sense. the fee structure should be low enough, just to prevent abuse.
Logged

LocoMB

  • Hero Member
  • *****
  • Karma: +101/-37
  • Offline Offline
  • Posts: 751
    • View Profile
Re: New fees and size limits in 1.7
« Reply #63 on: November 15, 2015, 05:47:03 am »

why not make the base fee changeable in the block-chain?
If nxt price go more than 10x higher, the fee is too high. We should prepare for the high price, not to limit the price to go high.

Before, the price is main determined by speculation when people just buy in a centralized exchange. Specutors do not care much about the transation fee, because it will not affect them much.
But we aim to make the price determined by application in future, if nxt will to success. Given this, we need to encourage usage of the our block chain, and lower fee make sense. the fee structure should be low enough, just to prevent abuse.

That is what I am worried about too! When volatility occurs, it changes the whole fee structure, and can seriously impede services that rely on a certain calculations!

How about this:
Pegging a base fee like a message or sendMoney TX to an average basket of some real world utility, for example 50th of a surface mail letter.
Such a physical utility fee is under physical cost constraints, because the letter has to be physically handled and expedited, and that is always related to resource and time usage.
PoW is linked to electricity costs- this scheme would create a somehwat similar link too.
As in this discussion the term 'utility value' keeps popping up, why not take it to the logical step and tie the base TX cost to a physical utility?

There is a nice example for an immutable indicator like this: The Baltic Dry Index (BDI)

This is the only index that cannot be manipulated, and there exist no futures and other leveraged instruments for this, hence the BDI is the best gauge for world trade.
Surface mail letter costs are similar immutable costs of transporting information stored on a physical medium, but it could also be any similar utility used as gauge.
« Last Edit: November 15, 2015, 05:53:18 am by LocoMB »
Logged
TOX
90E54E5B5213290EE616D425CADC473038CFABFA53C913271AA8559D1937DC4AF3A354A9E4E5

Jean-Luc

  • Core Dev
  • Hero Member
  • *****
  • Karma: +816/-81
  • Offline Offline
  • Posts: 1610
    • View Profile
Re: New fees and size limits in 1.7
« Reply #64 on: November 15, 2015, 06:06:57 am »

why not make the base fee changeable in the block-chain?
If nxt price go more than 10x higher, the fee is too high. We should prepare for the high price, not to limit the price to go high.
It has been said several times that if the price goes significantly higher, we will do another poll and change the fee, just like we are doing now. Price was already 10x higher once, with the same fee, and it was not a problem. We do hard forks about every 6 months, if the price starts going solidly up we do have time to react, and in the worst case of very sudden price increase we will do another hard fork just to adjust the fees sooner.

Historically, NXT has been less volatile than other cryptocurrencies.
 
Quote
How about this:
Pegging a base fee like a message or sendMoney TX to an average basket of some real world utility, for example 50th of a surface mail letter.
Such a physical utility fee is under physical cost constraints, because the letter has to be physically handled and expedited, and that is always related to resource and time usage.
PoW is linked to electricity costs- this scheme would create a somehwat similar link too.
As in this discussion the term 'utility value' keeps popping up, why not take it to the logical step and tie the base TX cost to a physical utility?

There is a nice example for an immutable indicator like this: The Baltic Dry Index (BDI)

This is the only index that cannot be manipulated, and there exist no futures and other leveraged instruments for this, hence the BDI is the best gauge for world trade.
Surface mail letter costs are similar immutable costs of transporting information stored on a physical medium, but it could also be any similar utility used as gauge.
How do you get the blockchain to read such an external index in a safe and trustless manner? How long will it take to implement and make sure it is secure? Is the complication and the risk of doing that automatically really worth it? This is a topic for a separate discussion, now we need to decide about the changes to take effect in 1.7, given the tools (voting) and software features we already have.
Logged
GPG key fingerprint: 263A 9EB0 29CF C77A 3D06  FD13 811D 6940 E1E4 240C
NXT-X4LF-9A4G-WN9Z-2R322

Jean-Luc

  • Core Dev
  • Hero Member
  • *****
  • Karma: +816/-81
  • Offline Offline
  • Posts: 1610
    • View Profile
Re: New fees and size limits in 1.7
« Reply #65 on: November 15, 2015, 06:33:26 am »

Prunable storage of people's deposit/withdraw info?
Yes, of course! This data is not needed for the blockchain to function, only your software needs it, therefore your software must take care of preserving it. And this has been made exceedingly easy for you, because you only need to change a few nxt.properties in the config file of your servers (servers you control, not public ones) to have them keep this prunable data forever for you, without forcing everyone else to also keep it forever.

Quote
That will open up a lot of new edge cases. I certainly dont want to rely on public servers keeping permanent copies of all such info and there is no guarantee that there would be any such servers with a permanent record
No it will not. And I said, you should run your own servers that you control. The public servers are for a backup - and you already rely on them to serve you the blockchain, right?

Quote
so the data needs to be in the permanent data. I keep saying this over and over

permanent data and it is part of the asset transfers, so there is no chance of them getting lost.
Such statements only show that you have no clue about what prunable data is.

Quote
I certainly dont have a month (yes, debugging MGW is very timeconsuming)
Yeah, a common sign of badly designed code.

Quote
to code and debug an MGW that uses prunable messages (using remote servers, setting up some internal servers as archive nodes, testing remote retrieval, etc). Even if I had the time, this prunable design would degrade MGW reliability by making it a LOT more complicated than now. Plus it would depend on public servers for proper operations
It has been explained to you that you can and should run your own servers, set to store prunable data forever, and as long as your servers don't all go down for more than two weeks at a time you never need to use public servers, and if your servers do all go down for more than two weeks, only then they will need to retrieve the expired prunable data from such public archival nodes. And this retrieval process has been all implemented and tested by the Nxt core developers, who have spent months on that so that projects like yours can afford to be unreliable, have weeks of downtime and keep no backups, yet still be able to restore their data from public archival nodes when needed.

Quote
The only practical solution i see if txfees go up is to increase the MGW fee, that is not a problem for me to do. I will just charge 10NXT txfee for each MGW tx, which would cover the worst case costs and a bit extra
You don't care about adding bloat to the Nxt blockchain, and now you confirm you don't care about your users either.
Logged
GPG key fingerprint: 263A 9EB0 29CF C77A 3D06  FD13 811D 6940 E1E4 240C
NXT-X4LF-9A4G-WN9Z-2R322

Jean-Luc

  • Core Dev
  • Hero Member
  • *****
  • Karma: +816/-81
  • Offline Offline
  • Posts: 1610
    • View Profile
Re: New fees and size limits in 1.7
« Reply #66 on: November 15, 2015, 06:46:25 am »

MGW uses the blockchain as its permanent record of deposits and withdraws. My understanding is that using prunable transactions means it disappears after 2 weeks.

How would you suggest to correlate the transactions made if the data is pruned away?
By setting nxt.maxPrunableLifetime=-1 and nxt.includeExpiredPrunable=true, your servers will keep prunable data forever, and serve it on request. At this point it becomes indistinguishable from permanent.

Quote
MGW recalculates balances by scanning the blockchain for all transactions, you know, like a blockchain.
No, it is using the blockchain as a database for storing arbitrary data. And this is not what the blockchain is designed for. The blockchain is a tool for achieving decentralized consensus, maintaining decentralized ledger, and distributing data between peers in a trustless and verifiable manner. Once such arbitrary data has been distributed, any part of it that is not required by the blockchain itself (i.e. by new peers to be able to recreate the blockchain state and reach consensus with others in a trustless way), does not need to be kept by all peers. Software that needs such data should keep it itself in its own database, or disable pruning locally to continue keeping it in the same database as the Nxt blockchain.

Quote
I am confused why you keep advocating MGW use prunable data. Where should I store the user transactions info. These are part of the deposit/withdraws so there is no chance of them being separated and lost. Once we start storing the user transaction data in any other place than the actual transaction, then it loses the blockchain aspect it has.
Already explained. Disable pruning locally, and the data will never be lost. If your servers all crash or are destroyed at once, such data is still retrievable from public archival nodes. Run a few archival nodes yourself in separate locations, or find a way to motivate people to run them by paying them with some asset.

For light clients, have them verify that the public node they connect to provides the prunable service, i.e. stores expired prunable data indefinitely. I think it already became clear that light clients must check version and other properties of public nodes they rely on.
Logged
GPG key fingerprint: 263A 9EB0 29CF C77A 3D06  FD13 811D 6940 E1E4 240C
NXT-X4LF-9A4G-WN9Z-2R322

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: New fees and size limits in 1.7
« Reply #67 on: November 15, 2015, 07:04:37 am »

Prunable storage of people's deposit/withdraw info?
Yes, of course! This data is not needed for the blockchain to function, only your software needs it, therefore your software must take care of preserving it. And this has been made exceedingly easy for you, because you only need to change a few nxt.properties in the config file of your servers (servers you control, not public ones) to have them keep this prunable data forever for you, without forcing everyone else to also keep it forever.

Quote
That will open up a lot of new edge cases. I certainly dont want to rely on public servers keeping permanent copies of all such info and there is no guarantee that there would be any such servers with a permanent record
No it will not. And I said, you should run your own servers that you control. The public servers are for a backup - and you already rely on them to serve you the blockchain, right?

Quote
so the data needs to be in the permanent data. I keep saying this over and over

permanent data and it is part of the asset transfers, so there is no chance of them getting lost.
Such statements only show that you have no clue about what prunable data is.

Quote
I certainly dont have a month (yes, debugging MGW is very timeconsuming)
Yeah, a common sign of badly designed code.

Quote
to code and debug an MGW that uses prunable messages (using remote servers, setting up some internal servers as archive nodes, testing remote retrieval, etc). Even if I had the time, this prunable design would degrade MGW reliability by making it a LOT more complicated than now. Plus it would depend on public servers for proper operations
It has been explained to you that you can and should run your own servers, set to store prunable data forever, and as long as your servers don't all go down for more than two weeks at a time you never need to use public servers, and if your servers do all go down for more than two weeks, only then they will need to retrieve the expired prunable data from such public archival nodes. And this retrieval process has been all implemented and tested by the Nxt core developers, who have spent months on that so that projects like yours can afford to be unreliable, have weeks of downtime and keep no backups, yet still be able to restore their data from public archival nodes when needed.

Quote
The only practical solution i see if txfees go up is to increase the MGW fee, that is not a problem for me to do. I will just charge 10NXT txfee for each MGW tx, which would cover the worst case costs and a bit extra
You don't care about adding bloat to the Nxt blockchain, and now you confirm you don't care about your users either.
Why are you making personal attacks on me? I think that is uncalled for.

BTC blockchain is quite large and each test requires to restart MGW, which scans the BTC data. That takes quite a bit of time. Since you are much better at creating well designed software than me, please tell me how I can make this go faster.

Since all MGW servers are running in datacenters, I dont really have control over them. Which means the user's mgw tx data is not protected by the decentralized NXT network.

MGW does not use public nodes.

And if to use public nodes, I would think there needs to be some contingency to deal with not being able to find the data. THAT is why I do not feel MGW using prunable data is wise. There are much fewer copies of the pruned data, since it is pruned. So a lot higher chance that it cant be found, to the point where I need to specifically deal with such cases.

Unless you are advocating that I just trust that there will always be public nodes available with a complete and accurate archive.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

chanc3r

  • Hero Member
  • *****
  • Karma: +124/-50
  • Offline Offline
  • Posts: 1019
  • NXTInspect
    • View Profile
Re: New fees and size limits in 1.7
« Reply #68 on: November 15, 2015, 09:50:32 am »

The public availability of an unabridged blockchain as we move to this new fees and limits model is a conversation that is driven by MGW / SuperNET now and I posted similar points in the 1.7 changelog thread.

We might not see other businesses talking about this yet and we should respect the utility that SuperNET has tried to drive into the NXT blockchain and hopefully many others will follow. If this happens then more people will consider and question the integrity of their ability to get a full blockchain copy.

The whole point of a blockchain based system is not to rely on data centre backups and shipping blockchain copies from one data centre to another using traditional methods.

If a prunable NXT is to succeed then a NXT business will need to be confident that it will always be able to access an unabridged copy of the blockchain and if the principles of decentralisation are to be maintained this cannot be solely the obligation of that business.

One of the things that it would be useful to be able to monitor is the health or contactable number of peers making the unabridged blockchain available, I guess a bit like the old torrent stuff, there may be lots of people who a sharing parts of the torrent but there are sometimes only a few sharing the complete torrent...

btw its not going to help if this descends into criticism / claims of how or why things are the way they are.
Logged
NXT: 29996814460165 (NXT-JTA7-B2QR-8BFC-2V222)
@imrimr @NXTinspect

cryptoscout

  • Hero Member
  • *****
  • Karma: +100/-96
  • Offline Offline
  • Posts: 635
    • View Profile
Re: New fees and size limits in 1.7
« Reply #69 on: November 15, 2015, 10:45:07 am »

Guys,

I've reading this for the past days, BUT?

Nobody has actually asked the real reason why the fee should be flexible, Secondly have you actually asked what projects and business really need?

Personally, why don't you approach different models and see what they are paying and what they could pay and take this from a commercial view opposed to just a technical view.

I have my own views on this for the mobile banking, Digital Identity and reward programs as my view is every transaction of hash needs to be on the blockchain.
Logged

Brangdon

  • Hero Member
  • *****
  • Karma: +229/-25
  • Offline Offline
  • Posts: 1389
  • Quality is addictive.
    • View Profile
Re: New fees and size limits in 1.7
« Reply #70 on: November 15, 2015, 10:51:17 am »

BTC blockchain is quite large and each test requires to restart MGW, which scans the BTC data. That takes quite a bit of time. Since you are much better at creating well designed software than me, please tell me how I can make this go faster.
Why can't you cache the state of the blockchain at some height, and then reload from that cache when you are debugging? A production restart would scan the entire blockchain as usual; this would just be a time-saver when testing. As a programmer myself, I know it's painful to have a long edit-compile-test cycle, and it is worth building extra infrastructure just to make it faster for debugging.

Quote
And if to use public nodes, I would think there needs to be some contingency to deal with not being able to find the data.
The suggestion is that you use private NRS nodes that you control.

(Although, if you made them public, that would help other companies that need archived data by providing some redundancy. Hopefully they will do the same. This needs to be bootstrapped.)

The whole point of a blockchain based system is not to rely on data centre backups and shipping blockchain copies from one data centre to another using traditional methods.
Once you have the archiving NRS nodes, you wouldn't need to be shipping blockchain copies using traditional methods. NRS does that itself. And, unlike a centralised data centre, it would be trust-free, because the Nxt block-chain stores hashes that ensure the nodes that store the archive don't change history.
Logged

Jean-Luc

  • Core Dev
  • Hero Member
  • *****
  • Karma: +816/-81
  • Offline Offline
  • Posts: 1610
    • View Profile
Re: New fees and size limits in 1.7
« Reply #71 on: November 15, 2015, 10:58:04 am »

have you actually asked what projects and business really need?

Personally, why don't you approach different models and see what they are paying and what they could pay and take this from a commercial view opposed to just a technical view.
Businesses want zero fees, isn't that obvious? Would you expect any business to say they want to pay a fee?

For those who think our fees are high enough to prevent spam, just look at the most recent blockchain transactions right now.

Quote
I have my own views on this for the mobile banking, Digital Identity and reward programs as my view is every transaction of hash needs to be on the blockchain.
This is nonsense. Only what the blockchain needs should be stored permanently on the blockchain. You can use the blockchain to verify the hashes of your data, that was at some point processed through it, but storing the actual data forever is not what the Nxt blockchain is designed for. Do it yourself, setup an archival node, pay someone to run an archival node for you, or use a coin that is specifically designed to provide distributed storage (sia, storj, maidsafe, etc).
Logged
GPG key fingerprint: 263A 9EB0 29CF C77A 3D06  FD13 811D 6940 E1E4 240C
NXT-X4LF-9A4G-WN9Z-2R322

EvilDave

  • Hero Member
  • *****
  • Karma: +341/-40
  • Offline Offline
  • Posts: 1789
    • View Profile
    • NXT Foundation
Re: New fees and size limits in 1.7
« Reply #72 on: November 15, 2015, 11:41:05 am »

Maybe it's time to simply acknowledge that there will be a need for Nxt archival nodes, and to start off a seperate discussion exploring how best to maintain (or encourage people to maintain) these archival nodes.
Logged
Nulli Dei, nulli Reges, solum NXT
NXT Donations: NXT-BNZB-9V8M-XRPW-3S3WD
We will ride eternal, shiny and chrome!

Damelon

  • Hero Member
  • *****
  • Karma: +792/-54
  • Offline Offline
  • Posts: 2314
    • View Profile
    • Nxt Inside
Re: New fees and size limits in 1.7
« Reply #73 on: November 15, 2015, 11:48:48 am »

have you actually asked what projects and business really need?

Personally, why don't you approach different models and see what they are paying and what they could pay and take this from a commercial view opposed to just a technical view.
Businesses want zero fees, isn't that obvious? Would you expect any business to say they want to pay a fee?

For those who think our fees are high enough to prevent spam, just look at the most recent blockchain transactions right now.

This is not completely true.

Business don't like fees is probably a more accurate statement, especially fees they don't see a use for.
Businesses do not mind fees that come with a benefit, and where it's clear that they get value for them.

I would not mind at all to pay fees that enable the party getting them to make sure my business benefits from them.

This is one of the core reasons sales and marketing exists: to explain how costs relate to services. If that translates into a fair deal, no business worth its salt will actually mind.
Logged
Member of the Nxt Foundation | Donations: NXT-D6K7-MLY6-98FM-FLL5T
Join Nxt Slack! https://nxtchat.herokuapp.com/
Founder of Blockchain Workspace | Personal Site & Blog

cc001

  • Hero Member
  • *****
  • Karma: +68/-4
  • Offline Offline
  • Posts: 829
    • View Profile
Re: New fees and size limits in 1.7
« Reply #74 on: November 15, 2015, 11:50:19 am »

Maybe it's time to simply acknowledge that there will be a need for Nxt archival nodes, and to start off a seperate discussion exploring how best to maintain (or encourage people to maintain) these archival nodes.

would it be possible to create a new/separate business branch on top of maintaining archival nodes? Similar to forgers or miners, you could earn some fees (or other kind of income) if you maintain such archival nodes?
Logged
cc001 Personal - NXT-8RXS-2SSK-RNF2-HSNL8
NxtReporting.com - The Nxt Asset Exchange Portfolio Manager with Profitability Tracking - Donations are greatly appreciated on NXT-5W4G-GAR6-JHJP-H8ZTW

Seccour

  • Sr. Member
  • ****
  • Karma: +68/-15
  • Offline Offline
  • Posts: 380
    • View Profile
Re: New fees and size limits in 1.7
« Reply #75 on: November 15, 2015, 12:22:53 pm »

Maybe it's time to simply acknowledge that there will be a need for Nxt archival nodes, and to start off a seperate discussion exploring how best to maintain (or encourage people to maintain) these archival nodes.

would it be possible to create a new/separate business branch on top of maintaining archival nodes? Similar to forgers or miners, you could earn some fees (or other kind of income) if you maintain such archival nodes?

Yeah but from where the income will come from ? And how we will know if the node realy archive everything ?
Logged
SecFund : 9125535795764729261 (Asset ID)

allwelder

  • Hero Member
  • *****
  • Karma: +196/-13
  • Offline Offline
  • Posts: 1867
  • NxtChina.org
    • View Profile
    • NxtChina.org
Re: New fees and size limits in 1.7
« Reply #76 on: November 15, 2015, 12:26:37 pm »

Low fees are killing forging economy and so just dangerous
But high fees may killing user base.
What we need is more fees with more users and more transactions,but not increasing fees of single transaction by decreasing users or transactions,right?
Logged
NxtChina |Weibo |Twitter Donation welcomed:NXT-APL9-66GU-K8LY-B3JJJ

cryptoscout

  • Hero Member
  • *****
  • Karma: +100/-96
  • Offline Offline
  • Posts: 635
    • View Profile
Re: New fees and size limits in 1.7
« Reply #77 on: November 15, 2015, 12:43:03 pm »

have you actually asked what projects and business really need?

Personally, why don't you approach different models and see what they are paying and what they could pay and take this from a commercial view opposed to just a technical view.
Businesses want zero fees, isn't that obvious? Would you expect any business to say they want to pay a fee?

For those who think our fees are high enough to prevent spam, just look at the most recent blockchain transactions right now.

Quote
I have my own views on this for the mobile banking, Digital Identity and reward programs as my view is every transaction of hash needs to be on the blockchain.
This is nonsense. Only what the blockchain needs should be stored permanently on the blockchain. You can use the blockchain to verify the hashes of your data, that was at some point processed through it, but storing the actual data forever is not what the Nxt blockchain is designed for. Do it yourself, setup an archival node, pay someone to run an archival node for you, or use a coin that is specifically designed to provide distributed storage (sia, storj, maidsafe, etc).


1st of all you don't need to store forever hence why Pruning is great some items may need to be stored 6 months to 7 years.

Secondly people don't want to pay fees but still have to in the real world and those doing a crypto business end up paying more then existing.

Companies that pay 2.50$ to 7.50$ fees would handsomely pay 25 to 75 cents fee if you can provide the same services or function.

So you say what you think and I will say what I think, it will never match but if you can find the middle then you have the solution to the Fees and capability, IE low to higher for Certain functions.

AE makes a lot of transaction that could be set to a % not fixed
MS could be set to flat fee of 0.1 to 0.5 NXT to make it more compelling to use.
Messaging with different variants of Capacity at different fee levels.
Voting a high fee to create the campaign and something low like 0.01 NXT as right now the voting system is more costly than existing systems ( you will say now it bloats the chain what if the voting is prunable upto 6 months later)

I could go on with loads of ways to commercialize NXT to be compelling  based on its functions
Logged

chanc3r

  • Hero Member
  • *****
  • Karma: +124/-50
  • Offline Offline
  • Posts: 1019
  • NXTInspect
    • View Profile
Re: New fees and size limits in 1.7
« Reply #78 on: November 15, 2015, 01:28:42 pm »

No one is disagreeing that different value transactions have different fee's
This is already in the change log
These primary transactions and their parameters are NOT PRUNED...

People are voting for the base fee not all fees

The debate is around the use of AMs and storage of them.
The change is that the DEFAULT ubiquitous availability of  permanent additional transaction information stored in messages is reduced from 1000 (currently) to 160 bytes.
 
Whilst I might debate the 160bytes being too low with J-L I do agree that...
NXT is not a storage blockchain - it can't be because people providing the storage receive NO fees
Even the chains that exist to provide storage do not put the storage in the chain itself.

The mistake is to have started NXT with a 1K block for an AM and not a 64byte block - at least then increasing it could not break anything. Although people would be complaining about bloat due to the increased message size probably.

Pruning creates simply 2 views of the chain, a full view and an abridged view.. If you don't change with your config you will get and share the abridged view. users that need the full view need to configure and share the full view of the chain.

The issue of data availability only arises if there is no peer on the network you can reach which has an unabridged copy of the chain. If there are plenty of businesses / power users / community members out there who run with the settings outlined by J-L then this problem should never arise.

Applications that create permanent messages longer than 160 bytes need to adapt to create prunable messages, a simply parameter.

Applications that simply READ messages need NO Changes the copy of NXT that they access just needs to be configured to download and share an unpruned copy of the blockchain.
Logged
NXT: 29996814460165 (NXT-JTA7-B2QR-8BFC-2V222)
@imrimr @NXTinspect

EvilDave

  • Hero Member
  • *****
  • Karma: +341/-40
  • Offline Offline
  • Posts: 1789
    • View Profile
    • NXT Foundation
Re: New fees and size limits in 1.7
« Reply #79 on: November 15, 2015, 02:05:55 pm »

Maybe it's time to simply acknowledge that there will be a need for Nxt archival nodes, and to start off a seperate discussion exploring how best to maintain (or encourage people to maintain) these archival nodes.

would it be possible to create a new/separate business branch on top of maintaining archival nodes? Similar to forgers or miners, you could earn some fees (or other kind of income) if you maintain such archival nodes?

Yeah but from where the income will come from ? And how we will know if the node really archive everything ?

That's the stuff that needs to be decided, guys.

@jl777 and Jean-Luc: do you two agree (  ;D ) on the need for archival nodes ?
If so, we can start making plans to support and maintain these nodes.

I'm happy to commit right now, personally, to running an archival node, in any case.
Logged
Nulli Dei, nulli Reges, solum NXT
NXT Donations: NXT-BNZB-9V8M-XRPW-3S3WD
We will ride eternal, shiny and chrome!

Seccour

  • Sr. Member
  • ****
  • Karma: +68/-15
  • Offline Offline
  • Posts: 380
    • View Profile
Re: New fees and size limits in 1.7
« Reply #80 on: November 15, 2015, 02:16:03 pm »

Maybe it's time to simply acknowledge that there will be a need for Nxt archival nodes, and to start off a seperate discussion exploring how best to maintain (or encourage people to maintain) these archival nodes.

would it be possible to create a new/separate business branch on top of maintaining archival nodes? Similar to forgers or miners, you could earn some fees (or other kind of income) if you maintain such archival nodes?

Yeah but from where the income will come from ? And how we will know if the node really archive everything ?

That's the stuff that needs to be decided, guys.

@jl777 and Jean-Luc: do you two agree (  ;D ) on the need for archival nodes ?
If so, we can start making plans to support and maintain these nodes.

I'm happy to commit right now, personally, to running an archival node, in any case.

I'm going to install 1 to 5 nodes for some of projects so i don't mind to make them to be archival node too
Logged
SecFund : 9125535795764729261 (Asset ID)

pazor

  • Full Member
  • ***
  • Karma: +6/-3
  • Offline Offline
  • Posts: 107
  • NXT-2XT3-3RP8-FHMJ-6ANVS
    • View Profile
Re: New fees and size limits in 1.7
« Reply #81 on: November 15, 2015, 02:23:41 pm »

the fee of 1 NXT is perfect.
Logged
NXT: 5507488987668838177
BTC: 12jiBjT2GSWYk2HwYdPqsQMuLqZ1br9D37

bidji29

  • Sr. Member
  • ****
  • Karma: +53/-11
  • Offline Offline
  • Posts: 250
    • View Profile
Re: New fees and size limits in 1.7
« Reply #82 on: November 15, 2015, 02:30:27 pm »

jl777 and jean-luc please..
It feel like mommy and daddy fighting.  :'(
Logged

Jimmy2011

  • Sr. Member
  • ****
  • Karma: +24/-19
  • Offline Offline
  • Posts: 329
    • View Profile
Re: New fees and size limits in 1.7
« Reply #83 on: November 15, 2015, 02:41:52 pm »

Is this service provider or data provider?
I can install some nodes and store it permanently!

meme should be here.

Logged
NXT-LX5G-L63N-ST8S-9LVZY

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: New fees and size limits in 1.7
« Reply #84 on: November 15, 2015, 02:59:02 pm »

If there are no bugs in the prunable system and there are no datacenter disasters and no unexpected edge cases arise, then in theory MGW can just switch over from permanent messages.

However, I am literally working 18hrs a day and behind on instantdex, pangea and peggy and I simply do not have any time to deal with getting another iteration of MGW done.

The idea of going from blockchain to store the data to the MGW nodes plus public archive nodes is an interesting idea and theoretically it is a step toward allowing scalability. However, I do care about customer funds and since I do not have the time to oversee any new revision, unless somebody else takes over the responsibility for MGW code, it will need to stay with permanent data and higher fees.

What most people dont understand is that each time I get involved with an incident, it delays my work in other areas. It is not simply a matter to spend X hours and then switch back, if my flow is broken, it might take hours or a day to get back to where I was. And a new MGW release cycle involves a couple weeks of testing, in which I need to be involved in.

I simply cannot do it all and I am getting a bit tired and was hoping to be able to lower my daily workings to 12hrs a day, not to increase it to 20hrs per day (which I cant sustain)

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

bidji29

  • Sr. Member
  • ****
  • Karma: +53/-11
  • Offline Offline
  • Posts: 250
    • View Profile
Re: New fees and size limits in 1.7
« Reply #85 on: November 15, 2015, 04:12:58 pm »

If there are no bugs in the prunable system and there are no datacenter disasters and no unexpected edge cases arise, then in theory MGW can just switch over from permanent messages.

However, I am literally working 18hrs a day and behind on instantdex, pangea and peggy and I simply do not have any time to deal with getting another iteration of MGW done.

The idea of going from blockchain to store the data to the MGW nodes plus public archive nodes is an interesting idea and theoretically it is a step toward allowing scalability. However, I do care about customer funds and since I do not have the time to oversee any new revision, unless somebody else takes over the responsibility for MGW code, it will need to stay with permanent data and higher fees.

What most people dont understand is that each time I get involved with an incident, it delays my work in other areas. It is not simply a matter to spend X hours and then switch back, if my flow is broken, it might take hours or a day to get back to where I was. And a new MGW release cycle involves a couple weeks of testing, in which I need to be involved in.

I simply cannot do it all and I am getting a bit tired and was hoping to be able to lower my daily workings to 12hrs a day, not to increase it to 20hrs per day (which I cant sustain)

James


Well, it's not the NXT dev fault if you charged yourself with too much work/project

That's unfortunate the changement recquire more work from you, but you put yourself in this, and NXT develoment can't revolve around your lack of disponibility.

We can't wait, we have to go forward.
Logged

blackyblack1

  • Hero Member
  • *****
  • Karma: +165/-82
  • Offline Offline
  • Posts: 1764
    • View Profile
Re: New fees and size limits in 1.7
« Reply #86 on: November 15, 2015, 04:50:17 pm »

I want also to note that you cannot have both encrypted and unencrypted prunnable message in one transaction. Somebody could be not aware of it.

I want to propose some features to facilitate way to 1.7:

1. Have a simple way to obtain resulting fee. I think the easiest way now is to create transaction without broadcasting and take the fee from the result Json.
2. Have a simple way to obtain message limit size.
3. Make sure that I can setup archival node any time and not a single prunnable message from the past is lost. Maybe have a snapshot from the devs with all prunnable messages available.
4. Add channel and tags fields to all transactions.
Logged

Jean-Luc

  • Core Dev
  • Hero Member
  • *****
  • Karma: +816/-81
  • Offline Offline
  • Posts: 1610
    • View Profile
Re: New fees and size limits in 1.7
« Reply #87 on: November 16, 2015, 07:55:41 am »

1. Have a simple way to obtain resulting fee. I think the easiest way now is to create transaction without broadcasting and take the fee from the result Json.
This is the simplest, and the only way possible. Since the fee can depend on many factors, and those factors can change in a future hard fork, the only reliable way is to create the transaction without signing and without broadcasting it, and have the server determine the minimum fee. This is how we do it in the UI too in 1.7.

Quote
2. Have a simple way to obtain message limit size.
It will be a constant available from getConstants.

Quote
3. Make sure that I can setup archival node any time and not a single prunnable message from the past is lost.
This is already possible with 1.6.2. If configured with nxt.maxPrunableLifetime=-1 and nxt.includeExpiredPrunable=true, the node will have pruning disabled, and will announce to others that it provides the PRUNABLE peer service. There are three ways to bootstrap such a node to have it download pruned messages from other archival nodes on the network - download the blockchain from scratch, do a full rescan, or use the new retrievePrunedData API available under Debug. Running this API will also tell you how many prunable data are still missing.

Quote

Maybe have a snapshot from the devs with all prunnable messages available.
A snapshot would be centralization, so not from the devs. But at least me and ScripterRon have been running our nodes set as archival for some time already, and they have all prunable data since the prunable feature was launched, nothing has been lost.

Quote

4. Add channel and tags fields to all transactions.
What for? This is adding more bloat, certainly not in 1.7. At this stage no new features can be added to 1.7 anyway.
Logged
GPG key fingerprint: 263A 9EB0 29CF C77A 3D06  FD13 811D 6940 E1E4 240C
NXT-X4LF-9A4G-WN9Z-2R322

blackyblack1

  • Hero Member
  • *****
  • Karma: +165/-82
  • Offline Offline
  • Posts: 1764
    • View Profile
Re: New fees and size limits in 1.7
« Reply #88 on: November 16, 2015, 09:01:23 am »

Quote

4. Add channel and tags fields to all transactions.
What for? This is adding more bloat, certainly not in 1.7. At this stage no new features can be added to 1.7 anyway.
You want to discourage using messages for services. Services are using messages to find own data and sort transactions based on it. Channels and tags solve this problem, discourage using messages and speed up database lookup.
Logged

Jean-Luc

  • Core Dev
  • Hero Member
  • *****
  • Karma: +816/-81
  • Offline Offline
  • Posts: 1610
    • View Profile
Re: New fees and size limits in 1.7
« Reply #89 on: November 16, 2015, 09:08:44 am »

You want to discourage using messages for services. Services are using messages to find own data and sort transactions based on it. Channels and tags solve this problem, discourage using messages and speed up database lookup.
So this is about adding metadata to transactions, in some more structured form than just arbitrary messages, indexed and searchable. I actually thought about this before, it is a bit similar to the account properties feature but for transactions.
It should be an optional attachment, not something that every transaction must have. We can design a new Appendix type (those can be attached to a transaction of any type, just like messages), containing one or more name/value pairs, and populate those in a transaction_properties table that can be queried.
I will keep that in mind as a feature request, but really no way to complete it for 1.7.

Edit:
And just like messages, those transaction properties should come in a permanent and prunable version, prunable costing substantially less. Long term, we would need configurability for finer pruning control, in this case for example a node that only cares about a particular metadata should prune transaction properties of all transactions unless they contain a specific tag. But such configurability can be added later, without even needing a hard fork, as long as the metadata has been created as prunable to begin with.


 
« Last Edit: November 16, 2015, 09:31:32 am by Jean-Luc »
Logged
GPG key fingerprint: 263A 9EB0 29CF C77A 3D06  FD13 811D 6940 E1E4 240C
NXT-X4LF-9A4G-WN9Z-2R322

Brangdon

  • Hero Member
  • *****
  • Karma: +229/-25
  • Offline Offline
  • Posts: 1389
  • Quality is addictive.
    • View Profile
Re: New fees and size limits in 1.7
« Reply #90 on: November 16, 2015, 11:01:22 am »

You want to discourage using messages for services. Services are using messages to find own data and sort transactions based on it. Channels and tags solve this problem, discourage using messages and speed up database lookup.
Can't they include their own data in the 32 bytes of free data they get with every transaction?

Having it structured might help performance, but if they are currently using messages, those aren't structured either.
Logged

blackyblack1

  • Hero Member
  • *****
  • Karma: +165/-82
  • Offline Offline
  • Posts: 1764
    • View Profile
Re: New fees and size limits in 1.7
« Reply #91 on: November 17, 2015, 08:04:07 pm »

Jean-Luc do we have any way to make sure that all prunnable messages are available on my node? So there is not a single chance to get empty message instead of full message. Does retrievePrunedData call give such guarantee?
Logged

Jean-Luc

  • Core Dev
  • Hero Member
  • *****
  • Karma: +816/-81
  • Offline Offline
  • Posts: 1610
    • View Profile
Re: New fees and size limits in 1.7
« Reply #92 on: November 17, 2015, 08:50:15 pm »

Jean-Luc do we have any way to make sure that all prunnable messages are available on my node? So there is not a single chance to get empty message instead of full message. Does retrievePrunedData call give such guarantee?
I should explain exactly how it works. The retrievePrunedData call goes through all prunable transactions, finds those that have been pruned, and populates their id's in a set, kept in memory. It returns immediately after that, returning the number of pruned transactions, i.e. how many have their prunable parts missing.

The thread that downloads the blockchain takes care of also scheduling a job that goes through all transaction id's in the above set, and retrieves them from archival nodes found on the network. If it cannot find and restore some of those, it waits for one hour before trying again. It will log debug messages to nxt.log about the number of prunable data restored, and the number still missing. As long as you don't shutdown the server, it will keep trying. You can watch the log to see when it has restored everything, or just call retrievePrunedData again and check the response, it if says numberOfPrunedData: 0, all have been restored. If you need to restart, before the restoration has completed, you should call retrievePrunedData again to populate that set of still missing transactions.

All the above only matters when you are bootstrapping a new node, or have had a downtime of more than two weeks. Once your node has restored pruned data older than two weeks, it is guaranteed that any new transactions received will be full, as this is the minimum expiration time enforced on all peers, transactions that are younger than two weeks but have their prunable parts missing are considered invalid and will not be accepted, and peers trying to send you such transactions will be blacklisted just like for any invalid transaction.

Also, if downloading the blockchain from scratch, or after a long downtime, you don't really need to call retrievePrunedData, the set of pruned transactions is populated again during the download and the restore task triggered automatically. It is only to force a restore task manually, e.g. for an already downloaded blockchain after changing the prunable lifetime setting, or after a restart that interrupted the restore, or to check the current number of missing pruned data to restore.

« Last Edit: November 17, 2015, 08:58:37 pm by Jean-Luc »
Logged
GPG key fingerprint: 263A 9EB0 29CF C77A 3D06  FD13 811D 6940 E1E4 240C
NXT-X4LF-9A4G-WN9Z-2R322

Brangdon

  • Hero Member
  • *****
  • Karma: +229/-25
  • Offline Offline
  • Posts: 1389
  • Quality is addictive.
    • View Profile
Re: New fees and size limits in 1.7
« Reply #93 on: November 18, 2015, 09:34:30 pm »

still figuring out the minimum needed to sustain MGW unchanged but 160 is definitely too low.
Have you figured this out yet? I'm curious to know how many bytes we're actually fighting over.
Logged

chanc3r

  • Hero Member
  • *****
  • Karma: +124/-50
  • Offline Offline
  • Posts: 1019
  • NXTInspect
    • View Profile
Re: New fees and size limits in 1.7
« Reply #94 on: November 20, 2015, 11:13:31 pm »

still figuring out the minimum needed to sustain MGW unchanged but 160 is definitely too low.
Have you figured this out yet? I'm curious to know how many bytes we're actually fighting over.

My script that scans for MGW asset transfers and checks the length of any message attachment has so far found a maximum of 197 so a permanent message block of 32x7 should eliminate the need for significant MGW changes.
« Last Edit: November 20, 2015, 11:20:40 pm by chanc3r »
Logged
NXT: 29996814460165 (NXT-JTA7-B2QR-8BFC-2V222)
@imrimr @NXTinspect

Brangdon

  • Hero Member
  • *****
  • Karma: +229/-25
  • Offline Offline
  • Posts: 1389
  • Quality is addictive.
    • View Profile
Re: New fees and size limits in 1.7
« Reply #95 on: November 21, 2015, 11:27:31 am »

Thanks. So a limit of 224 or 256 bytes would allow SuperNet to continue without breaking, albeit with higher transactions costs.

If the vote goes for "1 NXT or lower", would you still be motivated to switch SuperNet to using prunable attachments ASAP? Or could you put up with that cost indefinitely? I imagine the majority of your transactions would cost around 5 NXT.
Logged

chanc3r

  • Hero Member
  • *****
  • Karma: +124/-50
  • Offline Offline
  • Posts: 1019
  • NXTInspect
    • View Profile
Re: New fees and size limits in 1.7
« Reply #96 on: November 21, 2015, 12:05:12 pm »

Thanks. So a limit of 224 or 256 bytes would allow SuperNet to continue without breaking, albeit with higher transactions costs.

If the vote goes for "1 NXT or lower", would you still be motivated to switch SuperNet to using prunable attachments ASAP? Or could you put up with that cost indefinitely? I imagine the majority of your transactions would cost around 5 NXT.

Most of the discussion I've had with James is around ensuring security and availability of the data, not price..

The design of MGW is also decentralised, the idea being independent operators each running one of 'n' servers for each coin. Each server needs to guarantee the availability of a full blockchain, it should not depend on or have relationship with the other operators - this is important to ensure integrity of the m-of-n operating model to protect against fraud.
Also means if one operator is unreliable the others can find a new partner etc...

At the moment the preference is, if we have to, shorten the messages to fit within the limit because we know the messages are ubiquitously available still.
These are not stored separately for the reasons above - if you need to add a new member, they should bootstrap independently from the transaction/reconciliation data in the blockchain. This is at the core of James original design, taking a copy of a database from another MGW node or operator breaks the concept of independence.

So moving to prunable without clear establishment of a network of archive nodes which are independent of MGW operators is a  subtle breaking of some of the original core concepts behind creating a trustless, decentralised, blockchain based exchange.

At the moment with the state and reliability of archive service essentially unknown/unknowable? we have two choices, maintain compatibility with permanent messages or somehow bootstrap some independent archival nodes through SuperNET, depending on the scope and scale / reliability of the people doing this - this would also be a reduction in data resiliency/availability

So for now the option seems to be to try to make the messages fit, if the resulting transaction price is such that it prices MGW out of the market in the future then look at it again, call for a fee vote, consider other technical solutions or shut it down - as any operators of such a service need to be happy with the work and technical risk of running such a platform....
« Last Edit: November 21, 2015, 04:22:45 pm by chanc3r »
Logged
NXT: 29996814460165 (NXT-JTA7-B2QR-8BFC-2V222)
@imrimr @NXTinspect

Brangdon

  • Hero Member
  • *****
  • Karma: +229/-25
  • Offline Offline
  • Posts: 1389
  • Quality is addictive.
    • View Profile
Re: New fees and size limits in 1.7
« Reply #97 on: November 21, 2015, 03:45:39 pm »

The block-chain permanently stores a hash of the pruned data. You can use that hash to verify that the data provided by an archive node, or another operator's database, is the same as was originally stored. I can understand being concerned about pruned data being unobtainable, but not that the data might be tampered with. If you don't trust hashes, the whole block-chain concept fails.

Given that you don't have to trust the source of archival data, provided that its hashes match the block-chain, then finding a source becomes much easier.
Logged

chanc3r

  • Hero Member
  • *****
  • Karma: +124/-50
  • Offline Offline
  • Posts: 1019
  • NXTInspect
    • View Profile
Re: New fees and size limits in 1.7
« Reply #98 on: November 21, 2015, 04:46:54 pm »

The block-chain permanently stores a hash of the pruned data. You can use that hash to verify that the data provided by an archive node, or another operator's database, is the same as was originally stored. I can understand being concerned about pruned data being unobtainable, but not that the data might be tampered with. If you don't trust hashes, the whole block-chain concept fails.

Given that you don't have to trust the source of archival data, provided that its hashes match the block-chain, then finding a source becomes much easier.

I already understand that.

Unless there are always a number of nodes running with a lifetime of -1 then there is a finite risk prunable data disappears forever.. This is fine and I understand J-Ls concepts around discardable data and like the solution along with a lot of the other cool stuff in NXT.

MGW transaction records, however at the moment are not discardable so I would rather have them in the transaction table along with the rest of the transaction..

This is essentially a risk based scenario, in this case its not the developers but the operators of nodes who take the risks in case data goes permanently missing..

My experience is not all peers see all other peers so its possible one archive node will not see enough to get the blocks and there is again a risk it will end up with gaps. It also means making sure there are checks in code for pruned messages which have not downloaded for some reason along with handling logic.

Hopefully there will be a good number of archive nodes etc.. but at the moment the health of an archive copy of NXT is unclear and at the moment no way I know to determine it unless I know the operator of such a node or have one myself which I know has every piece of data.

If I have to ship data copies separately and then verify it with custom software against the blockchain hashes, typically the answer to this would be to find some software that does it for me, in this case it would look a lot like a different blockchain.
Logged
NXT: 29996814460165 (NXT-JTA7-B2QR-8BFC-2V222)
@imrimr @NXTinspect

blackyblack1

  • Hero Member
  • *****
  • Karma: +165/-82
  • Offline Offline
  • Posts: 1764
    • View Profile
Re: New fees and size limits in 1.7
« Reply #99 on: November 21, 2015, 05:28:17 pm »

Prunable data storage is basically the same as for permanent messages but with lower redundancy. You can estimate amount of archival nodes as order of magnitude less than normal nodes. So for each 10 nodes we are having 1 archival node.
If you think that amount of archival nodes is too small you could try to incentivize node owners to maintain archival node eg paying some reward monthly.
And there is an API call to check for gaps in prunnable messages. Use it to make sure your machine is ready to work as MGW server.
Logged

Brangdon

  • Hero Member
  • *****
  • Karma: +229/-25
  • Offline Offline
  • Posts: 1389
  • Quality is addictive.
    • View Profile
Re: New fees and size limits in 1.7
« Reply #100 on: November 21, 2015, 09:02:13 pm »

Unless there are always a number of nodes running with a lifetime of -1 then there is a finite risk prunable data disappears forever..
I think the intention is that there will always be some archival nodes. You can help ensure that by running archival nodes yourself. Each of your server operators can run one, so you have redundancy even within the SuperNet group.

Quote
Hopefully there will be a good number of archive nodes etc.. but at the moment the health of an archive copy of NXT is unclear and at the moment no way I know to determine it
It's available for a given node with getState. I have asked for this information to be included in the client GUI peer list, and hopefully other peer explorers will start including it too. I know I run an archival node, and the devs said they did, and EvilDave said he was going to. I don't see a problem with having semi-official ones with hallmarks from community funds.

Quote
If I have to ship data copies separately and then verify it with custom software against the blockchain hashes, typically the answer to this would be to find some software that does it for me, in this case it would look a lot like a different blockchain.
I'm sorry that I was pointing out the obvious earlier; but then you say things like this, which I'm struggling to understand. Why would you need to ship data separately? If you run a node, and make sure your other nodes can see it, the NRS itself will fetch the data from it and verify the hashes. The "software that does it for you" is NRS.
Logged

xiahui135

  • Full Member
  • ***
  • Karma: +15/-11
  • Offline Offline
  • Posts: 150
    • View Profile
Re: New fees and size limits in 1.7
« Reply #101 on: November 22, 2015, 09:35:40 am »

It has been said several times that if the price goes significantly higher, we will do another poll and change the fee, just like we are doing now. Price was already 10x higher once, with the same fee, and it was not a problem. We do hard forks about every 6 months, if the price starts going solidly up we do have time to react, and in the worst case of very sudden price increase we will do another hard fork just to adjust the fees sooner.
Historically, NXT has been less volatile than other cryptocurrencies.
We just do not want another fork. It hurts.
Logged

sadface

  • Sr. Member
  • ****
  • Karma: +16/-2
  • Offline Offline
  • Posts: 273
    • View Profile
Re: New fees and size limits in 1.7
« Reply #102 on: November 22, 2015, 10:29:56 am »

regarding the post above me: has there been a discussion about what a fee could be or look like in the possibly distant future? sometimes things go very quickly in the crypto world and saying we're going to do another fork when the time comes isn't really a good answer. looking at the controversy regarding fees right now we might have a hard time rushing a hard fork to adjust the fees and depending on the price development another and then another etc
Logged

Brangdon

  • Hero Member
  • *****
  • Karma: +229/-25
  • Offline Offline
  • Posts: 1389
  • Quality is addictive.
    • View Profile
Re: New fees and size limits in 1.7
« Reply #103 on: November 22, 2015, 03:01:23 pm »

The recent controversy hasn't been over fees levels as such. It's been over forcing app developers to switch to pruneable data, with the qualitative change in fee architecture one tool to encourage that. Lower size limits being another.

If everyone is following best practice, that is, using prunable data and setting the fees to 0 so the NRS calculates them, then future fee changes shouldn't cause app developers any coding work. Especially if the change is to reduce the fees rather than raise them.
Logged

chanc3r

  • Hero Member
  • *****
  • Karma: +124/-50
  • Offline Offline
  • Posts: 1019
  • NXTInspect
    • View Profile
Re: New fees and size limits in 1.7
« Reply #104 on: November 22, 2015, 10:17:18 pm »

@J-L - another one if I may...
Some characters in a JSON string need to be escaped,...
well " specifically.
so if I use a " in an AM - this will get encoded as \" in the output but I assume its 1 byte in the blockchain.

the question is - does this count as 1 character or two within the 160 limit?

and when does fee=0 take effect to mean 'use minimum fee' - is this already in place or only with 1.7?
« Last Edit: November 22, 2015, 10:55:51 pm by chanc3r »
Logged
NXT: 29996814460165 (NXT-JTA7-B2QR-8BFC-2V222)
@imrimr @NXTinspect

Jean-Luc

  • Core Dev
  • Hero Member
  • *****
  • Karma: +816/-81
  • Offline Offline
  • Posts: 1610
    • View Profile
Re: New fees and size limits in 1.7
« Reply #105 on: November 23, 2015, 09:38:12 am »

@J-L - another one if I may...
Some characters in a JSON string need to be escaped,...
well " specifically.
so if I use a " in an AM - this will get encoded as \" in the output but I assume its 1 byte in the blockchain.

the question is - does this count as 1 character or two within the 160 limit?
What matters is how it is stored in the blockchain, the escaping is in the output only. A " will take 1 byte, i.e. a message consisting of 32 " characters will fit exactly in the free limit. For permanent messages, which can be either text or binary, the size limits are currently based on their length in bytes; for other attachments such as aliases, that are text only, the limits are currently based on their character count. I see now that for aliases, texts in languages using double-byte characters will take more space than what they are being charged for, but I don't think it matters that much, those should really be a minority.

Quote
and when does fee=0 take effect to mean 'use minimum fee' - is this already in place or only with 1.7?
This has been in place since 1.5, maybe even 1.4. And this is why I keep saying that all changes needed to prepare for 1.7 can be done now, there is no need to wait for 1.7 stable.
Logged
GPG key fingerprint: 263A 9EB0 29CF C77A 3D06  FD13 811D 6940 E1E4 240C
NXT-X4LF-9A4G-WN9Z-2R322

MrCluster87

  • Hero Member
  • *****
  • Karma: +81/-3
  • Offline Offline
  • Posts: 855
    • View Profile
    • youtube
Re: New fees and size limits in 1.7
« Reply #106 on: January 09, 2016, 05:54:13 pm »

Nxt Arbitrary Messages use case: Certified e-mails: https://www.youtube.com/watch?v=ycPv0eFJw-k

yassin54

  • Hero Member
  • *****
  • Karma: +240/-14
  • Offline Offline
  • Posts: 2503
  • I am Homer, Sorry my english is Bad!!
    • View Profile
Pages: 1 2 3 ... 6 [All]
 

elective-stereophonic
elective-stereophonic
assembly
assembly