elective-stereophonic
elective-stereophonic
New fees and size limits in 1.7 singapore
Please login or register.

Login with username, password and session length
Advanced search  

News:

Latest Stable Nxt Client: Nxt 1.12.1 Upgrade before block 2870000 is mandatory!

Pages: 1 ... 3 4 [5] 6  All

Author Topic: New fees and size limits in 1.7  (Read 21474 times)

Seccour

  • Sr. Member
  • ****
  • Karma: +68/-15
  • Offline Offline
  • Posts: 380
    • View Profile
Re: New fees and size limits in 1.7
« Reply #80 on: November 15, 2015, 02:16:03 pm »

Maybe it's time to simply acknowledge that there will be a need for Nxt archival nodes, and to start off a seperate discussion exploring how best to maintain (or encourage people to maintain) these archival nodes.

would it be possible to create a new/separate business branch on top of maintaining archival nodes? Similar to forgers or miners, you could earn some fees (or other kind of income) if you maintain such archival nodes?

Yeah but from where the income will come from ? And how we will know if the node really archive everything ?

That's the stuff that needs to be decided, guys.

@jl777 and Jean-Luc: do you two agree (  ;D ) on the need for archival nodes ?
If so, we can start making plans to support and maintain these nodes.

I'm happy to commit right now, personally, to running an archival node, in any case.

I'm going to install 1 to 5 nodes for some of projects so i don't mind to make them to be archival node too
Logged
SecFund : 9125535795764729261 (Asset ID)

pazor

  • Full Member
  • ***
  • Karma: +6/-3
  • Offline Offline
  • Posts: 107
  • NXT-2XT3-3RP8-FHMJ-6ANVS
    • View Profile
Re: New fees and size limits in 1.7
« Reply #81 on: November 15, 2015, 02:23:41 pm »

the fee of 1 NXT is perfect.
Logged
NXT: 5507488987668838177
BTC: 12jiBjT2GSWYk2HwYdPqsQMuLqZ1br9D37

bidji29

  • Sr. Member
  • ****
  • Karma: +53/-11
  • Offline Offline
  • Posts: 250
    • View Profile
Re: New fees and size limits in 1.7
« Reply #82 on: November 15, 2015, 02:30:27 pm »

jl777 and jean-luc please..
It feel like mommy and daddy fighting.  :'(
Logged

Jimmy2011

  • Sr. Member
  • ****
  • Karma: +24/-19
  • Offline Offline
  • Posts: 329
    • View Profile
Re: New fees and size limits in 1.7
« Reply #83 on: November 15, 2015, 02:41:52 pm »

Is this service provider or data provider?
I can install some nodes and store it permanently!

meme should be here.

Logged
NXT-LX5G-L63N-ST8S-9LVZY

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: New fees and size limits in 1.7
« Reply #84 on: November 15, 2015, 02:59:02 pm »

If there are no bugs in the prunable system and there are no datacenter disasters and no unexpected edge cases arise, then in theory MGW can just switch over from permanent messages.

However, I am literally working 18hrs a day and behind on instantdex, pangea and peggy and I simply do not have any time to deal with getting another iteration of MGW done.

The idea of going from blockchain to store the data to the MGW nodes plus public archive nodes is an interesting idea and theoretically it is a step toward allowing scalability. However, I do care about customer funds and since I do not have the time to oversee any new revision, unless somebody else takes over the responsibility for MGW code, it will need to stay with permanent data and higher fees.

What most people dont understand is that each time I get involved with an incident, it delays my work in other areas. It is not simply a matter to spend X hours and then switch back, if my flow is broken, it might take hours or a day to get back to where I was. And a new MGW release cycle involves a couple weeks of testing, in which I need to be involved in.

I simply cannot do it all and I am getting a bit tired and was hoping to be able to lower my daily workings to 12hrs a day, not to increase it to 20hrs per day (which I cant sustain)

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

bidji29

  • Sr. Member
  • ****
  • Karma: +53/-11
  • Offline Offline
  • Posts: 250
    • View Profile
Re: New fees and size limits in 1.7
« Reply #85 on: November 15, 2015, 04:12:58 pm »

If there are no bugs in the prunable system and there are no datacenter disasters and no unexpected edge cases arise, then in theory MGW can just switch over from permanent messages.

However, I am literally working 18hrs a day and behind on instantdex, pangea and peggy and I simply do not have any time to deal with getting another iteration of MGW done.

The idea of going from blockchain to store the data to the MGW nodes plus public archive nodes is an interesting idea and theoretically it is a step toward allowing scalability. However, I do care about customer funds and since I do not have the time to oversee any new revision, unless somebody else takes over the responsibility for MGW code, it will need to stay with permanent data and higher fees.

What most people dont understand is that each time I get involved with an incident, it delays my work in other areas. It is not simply a matter to spend X hours and then switch back, if my flow is broken, it might take hours or a day to get back to where I was. And a new MGW release cycle involves a couple weeks of testing, in which I need to be involved in.

I simply cannot do it all and I am getting a bit tired and was hoping to be able to lower my daily workings to 12hrs a day, not to increase it to 20hrs per day (which I cant sustain)

James


Well, it's not the NXT dev fault if you charged yourself with too much work/project

That's unfortunate the changement recquire more work from you, but you put yourself in this, and NXT develoment can't revolve around your lack of disponibility.

We can't wait, we have to go forward.
Logged

blackyblack1

  • Hero Member
  • *****
  • Karma: +165/-82
  • Offline Offline
  • Posts: 1764
    • View Profile
Re: New fees and size limits in 1.7
« Reply #86 on: November 15, 2015, 04:50:17 pm »

I want also to note that you cannot have both encrypted and unencrypted prunnable message in one transaction. Somebody could be not aware of it.

I want to propose some features to facilitate way to 1.7:

1. Have a simple way to obtain resulting fee. I think the easiest way now is to create transaction without broadcasting and take the fee from the result Json.
2. Have a simple way to obtain message limit size.
3. Make sure that I can setup archival node any time and not a single prunnable message from the past is lost. Maybe have a snapshot from the devs with all prunnable messages available.
4. Add channel and tags fields to all transactions.
Logged

Jean-Luc

  • Core Dev
  • Hero Member
  • *****
  • Karma: +816/-81
  • Offline Offline
  • Posts: 1610
    • View Profile
Re: New fees and size limits in 1.7
« Reply #87 on: November 16, 2015, 07:55:41 am »

1. Have a simple way to obtain resulting fee. I think the easiest way now is to create transaction without broadcasting and take the fee from the result Json.
This is the simplest, and the only way possible. Since the fee can depend on many factors, and those factors can change in a future hard fork, the only reliable way is to create the transaction without signing and without broadcasting it, and have the server determine the minimum fee. This is how we do it in the UI too in 1.7.

Quote
2. Have a simple way to obtain message limit size.
It will be a constant available from getConstants.

Quote
3. Make sure that I can setup archival node any time and not a single prunnable message from the past is lost.
This is already possible with 1.6.2. If configured with nxt.maxPrunableLifetime=-1 and nxt.includeExpiredPrunable=true, the node will have pruning disabled, and will announce to others that it provides the PRUNABLE peer service. There are three ways to bootstrap such a node to have it download pruned messages from other archival nodes on the network - download the blockchain from scratch, do a full rescan, or use the new retrievePrunedData API available under Debug. Running this API will also tell you how many prunable data are still missing.

Quote

Maybe have a snapshot from the devs with all prunnable messages available.
A snapshot would be centralization, so not from the devs. But at least me and ScripterRon have been running our nodes set as archival for some time already, and they have all prunable data since the prunable feature was launched, nothing has been lost.

Quote

4. Add channel and tags fields to all transactions.
What for? This is adding more bloat, certainly not in 1.7. At this stage no new features can be added to 1.7 anyway.
Logged
GPG key fingerprint: 263A 9EB0 29CF C77A 3D06  FD13 811D 6940 E1E4 240C
NXT-X4LF-9A4G-WN9Z-2R322

blackyblack1

  • Hero Member
  • *****
  • Karma: +165/-82
  • Offline Offline
  • Posts: 1764
    • View Profile
Re: New fees and size limits in 1.7
« Reply #88 on: November 16, 2015, 09:01:23 am »

Quote

4. Add channel and tags fields to all transactions.
What for? This is adding more bloat, certainly not in 1.7. At this stage no new features can be added to 1.7 anyway.
You want to discourage using messages for services. Services are using messages to find own data and sort transactions based on it. Channels and tags solve this problem, discourage using messages and speed up database lookup.
Logged

Jean-Luc

  • Core Dev
  • Hero Member
  • *****
  • Karma: +816/-81
  • Offline Offline
  • Posts: 1610
    • View Profile
Re: New fees and size limits in 1.7
« Reply #89 on: November 16, 2015, 09:08:44 am »

You want to discourage using messages for services. Services are using messages to find own data and sort transactions based on it. Channels and tags solve this problem, discourage using messages and speed up database lookup.
So this is about adding metadata to transactions, in some more structured form than just arbitrary messages, indexed and searchable. I actually thought about this before, it is a bit similar to the account properties feature but for transactions.
It should be an optional attachment, not something that every transaction must have. We can design a new Appendix type (those can be attached to a transaction of any type, just like messages), containing one or more name/value pairs, and populate those in a transaction_properties table that can be queried.
I will keep that in mind as a feature request, but really no way to complete it for 1.7.

Edit:
And just like messages, those transaction properties should come in a permanent and prunable version, prunable costing substantially less. Long term, we would need configurability for finer pruning control, in this case for example a node that only cares about a particular metadata should prune transaction properties of all transactions unless they contain a specific tag. But such configurability can be added later, without even needing a hard fork, as long as the metadata has been created as prunable to begin with.


 
« Last Edit: November 16, 2015, 09:31:32 am by Jean-Luc »
Logged
GPG key fingerprint: 263A 9EB0 29CF C77A 3D06  FD13 811D 6940 E1E4 240C
NXT-X4LF-9A4G-WN9Z-2R322

Brangdon

  • Hero Member
  • *****
  • Karma: +229/-25
  • Offline Offline
  • Posts: 1389
  • Quality is addictive.
    • View Profile
Re: New fees and size limits in 1.7
« Reply #90 on: November 16, 2015, 11:01:22 am »

You want to discourage using messages for services. Services are using messages to find own data and sort transactions based on it. Channels and tags solve this problem, discourage using messages and speed up database lookup.
Can't they include their own data in the 32 bytes of free data they get with every transaction?

Having it structured might help performance, but if they are currently using messages, those aren't structured either.
Logged

blackyblack1

  • Hero Member
  • *****
  • Karma: +165/-82
  • Offline Offline
  • Posts: 1764
    • View Profile
Re: New fees and size limits in 1.7
« Reply #91 on: November 17, 2015, 08:04:07 pm »

Jean-Luc do we have any way to make sure that all prunnable messages are available on my node? So there is not a single chance to get empty message instead of full message. Does retrievePrunedData call give such guarantee?
Logged

Jean-Luc

  • Core Dev
  • Hero Member
  • *****
  • Karma: +816/-81
  • Offline Offline
  • Posts: 1610
    • View Profile
Re: New fees and size limits in 1.7
« Reply #92 on: November 17, 2015, 08:50:15 pm »

Jean-Luc do we have any way to make sure that all prunnable messages are available on my node? So there is not a single chance to get empty message instead of full message. Does retrievePrunedData call give such guarantee?
I should explain exactly how it works. The retrievePrunedData call goes through all prunable transactions, finds those that have been pruned, and populates their id's in a set, kept in memory. It returns immediately after that, returning the number of pruned transactions, i.e. how many have their prunable parts missing.

The thread that downloads the blockchain takes care of also scheduling a job that goes through all transaction id's in the above set, and retrieves them from archival nodes found on the network. If it cannot find and restore some of those, it waits for one hour before trying again. It will log debug messages to nxt.log about the number of prunable data restored, and the number still missing. As long as you don't shutdown the server, it will keep trying. You can watch the log to see when it has restored everything, or just call retrievePrunedData again and check the response, it if says numberOfPrunedData: 0, all have been restored. If you need to restart, before the restoration has completed, you should call retrievePrunedData again to populate that set of still missing transactions.

All the above only matters when you are bootstrapping a new node, or have had a downtime of more than two weeks. Once your node has restored pruned data older than two weeks, it is guaranteed that any new transactions received will be full, as this is the minimum expiration time enforced on all peers, transactions that are younger than two weeks but have their prunable parts missing are considered invalid and will not be accepted, and peers trying to send you such transactions will be blacklisted just like for any invalid transaction.

Also, if downloading the blockchain from scratch, or after a long downtime, you don't really need to call retrievePrunedData, the set of pruned transactions is populated again during the download and the restore task triggered automatically. It is only to force a restore task manually, e.g. for an already downloaded blockchain after changing the prunable lifetime setting, or after a restart that interrupted the restore, or to check the current number of missing pruned data to restore.

« Last Edit: November 17, 2015, 08:58:37 pm by Jean-Luc »
Logged
GPG key fingerprint: 263A 9EB0 29CF C77A 3D06  FD13 811D 6940 E1E4 240C
NXT-X4LF-9A4G-WN9Z-2R322

Brangdon

  • Hero Member
  • *****
  • Karma: +229/-25
  • Offline Offline
  • Posts: 1389
  • Quality is addictive.
    • View Profile
Re: New fees and size limits in 1.7
« Reply #93 on: November 18, 2015, 09:34:30 pm »

still figuring out the minimum needed to sustain MGW unchanged but 160 is definitely too low.
Have you figured this out yet? I'm curious to know how many bytes we're actually fighting over.
Logged

chanc3r

  • Hero Member
  • *****
  • Karma: +124/-50
  • Offline Offline
  • Posts: 1019
  • NXTInspect
    • View Profile
Re: New fees and size limits in 1.7
« Reply #94 on: November 20, 2015, 11:13:31 pm »

still figuring out the minimum needed to sustain MGW unchanged but 160 is definitely too low.
Have you figured this out yet? I'm curious to know how many bytes we're actually fighting over.

My script that scans for MGW asset transfers and checks the length of any message attachment has so far found a maximum of 197 so a permanent message block of 32x7 should eliminate the need for significant MGW changes.
« Last Edit: November 20, 2015, 11:20:40 pm by chanc3r »
Logged
NXT: 29996814460165 (NXT-JTA7-B2QR-8BFC-2V222)
@imrimr @NXTinspect

Brangdon

  • Hero Member
  • *****
  • Karma: +229/-25
  • Offline Offline
  • Posts: 1389
  • Quality is addictive.
    • View Profile
Re: New fees and size limits in 1.7
« Reply #95 on: November 21, 2015, 11:27:31 am »

Thanks. So a limit of 224 or 256 bytes would allow SuperNet to continue without breaking, albeit with higher transactions costs.

If the vote goes for "1 NXT or lower", would you still be motivated to switch SuperNet to using prunable attachments ASAP? Or could you put up with that cost indefinitely? I imagine the majority of your transactions would cost around 5 NXT.
Logged

chanc3r

  • Hero Member
  • *****
  • Karma: +124/-50
  • Offline Offline
  • Posts: 1019
  • NXTInspect
    • View Profile
Re: New fees and size limits in 1.7
« Reply #96 on: November 21, 2015, 12:05:12 pm »

Thanks. So a limit of 224 or 256 bytes would allow SuperNet to continue without breaking, albeit with higher transactions costs.

If the vote goes for "1 NXT or lower", would you still be motivated to switch SuperNet to using prunable attachments ASAP? Or could you put up with that cost indefinitely? I imagine the majority of your transactions would cost around 5 NXT.

Most of the discussion I've had with James is around ensuring security and availability of the data, not price..

The design of MGW is also decentralised, the idea being independent operators each running one of 'n' servers for each coin. Each server needs to guarantee the availability of a full blockchain, it should not depend on or have relationship with the other operators - this is important to ensure integrity of the m-of-n operating model to protect against fraud.
Also means if one operator is unreliable the others can find a new partner etc...

At the moment the preference is, if we have to, shorten the messages to fit within the limit because we know the messages are ubiquitously available still.
These are not stored separately for the reasons above - if you need to add a new member, they should bootstrap independently from the transaction/reconciliation data in the blockchain. This is at the core of James original design, taking a copy of a database from another MGW node or operator breaks the concept of independence.

So moving to prunable without clear establishment of a network of archive nodes which are independent of MGW operators is a  subtle breaking of some of the original core concepts behind creating a trustless, decentralised, blockchain based exchange.

At the moment with the state and reliability of archive service essentially unknown/unknowable? we have two choices, maintain compatibility with permanent messages or somehow bootstrap some independent archival nodes through SuperNET, depending on the scope and scale / reliability of the people doing this - this would also be a reduction in data resiliency/availability

So for now the option seems to be to try to make the messages fit, if the resulting transaction price is such that it prices MGW out of the market in the future then look at it again, call for a fee vote, consider other technical solutions or shut it down - as any operators of such a service need to be happy with the work and technical risk of running such a platform....
« Last Edit: November 21, 2015, 04:22:45 pm by chanc3r »
Logged
NXT: 29996814460165 (NXT-JTA7-B2QR-8BFC-2V222)
@imrimr @NXTinspect

Brangdon

  • Hero Member
  • *****
  • Karma: +229/-25
  • Offline Offline
  • Posts: 1389
  • Quality is addictive.
    • View Profile
Re: New fees and size limits in 1.7
« Reply #97 on: November 21, 2015, 03:45:39 pm »

The block-chain permanently stores a hash of the pruned data. You can use that hash to verify that the data provided by an archive node, or another operator's database, is the same as was originally stored. I can understand being concerned about pruned data being unobtainable, but not that the data might be tampered with. If you don't trust hashes, the whole block-chain concept fails.

Given that you don't have to trust the source of archival data, provided that its hashes match the block-chain, then finding a source becomes much easier.
Logged

chanc3r

  • Hero Member
  • *****
  • Karma: +124/-50
  • Offline Offline
  • Posts: 1019
  • NXTInspect
    • View Profile
Re: New fees and size limits in 1.7
« Reply #98 on: November 21, 2015, 04:46:54 pm »

The block-chain permanently stores a hash of the pruned data. You can use that hash to verify that the data provided by an archive node, or another operator's database, is the same as was originally stored. I can understand being concerned about pruned data being unobtainable, but not that the data might be tampered with. If you don't trust hashes, the whole block-chain concept fails.

Given that you don't have to trust the source of archival data, provided that its hashes match the block-chain, then finding a source becomes much easier.

I already understand that.

Unless there are always a number of nodes running with a lifetime of -1 then there is a finite risk prunable data disappears forever.. This is fine and I understand J-Ls concepts around discardable data and like the solution along with a lot of the other cool stuff in NXT.

MGW transaction records, however at the moment are not discardable so I would rather have them in the transaction table along with the rest of the transaction..

This is essentially a risk based scenario, in this case its not the developers but the operators of nodes who take the risks in case data goes permanently missing..

My experience is not all peers see all other peers so its possible one archive node will not see enough to get the blocks and there is again a risk it will end up with gaps. It also means making sure there are checks in code for pruned messages which have not downloaded for some reason along with handling logic.

Hopefully there will be a good number of archive nodes etc.. but at the moment the health of an archive copy of NXT is unclear and at the moment no way I know to determine it unless I know the operator of such a node or have one myself which I know has every piece of data.

If I have to ship data copies separately and then verify it with custom software against the blockchain hashes, typically the answer to this would be to find some software that does it for me, in this case it would look a lot like a different blockchain.
Logged
NXT: 29996814460165 (NXT-JTA7-B2QR-8BFC-2V222)
@imrimr @NXTinspect

blackyblack1

  • Hero Member
  • *****
  • Karma: +165/-82
  • Offline Offline
  • Posts: 1764
    • View Profile
Re: New fees and size limits in 1.7
« Reply #99 on: November 21, 2015, 05:28:17 pm »

Prunable data storage is basically the same as for permanent messages but with lower redundancy. You can estimate amount of archival nodes as order of magnitude less than normal nodes. So for each 10 nodes we are having 1 archival node.
If you think that amount of archival nodes is too small you could try to incentivize node owners to maintain archival node eg paying some reward monthly.
And there is an API call to check for gaps in prunnable messages. Use it to make sure your machine is ready to work as MGW server.
Logged
Pages: 1 ... 3 4 [5] 6  All
 

elective-stereophonic
elective-stereophonic
assembly
assembly