Switch to ArdorForum

Quote from: _mr_e on June 14, 2015, 08:18:00 pm

Quote from: verymuchso on June 14, 2015, 09:23:13 am

Quote from: _mr_e on June 12, 2015, 08:30:53 pm

So why don't you see your own unconfirmed order getting added to the order book when you place it anymore? I hope this wasn't a purposeful change. It's especially annoying when using jnxt.org/nxt in combination with nxtvault or jay because you no longer have any feedback of your order going through!

Because that's a very difficult thing to do (because of how the data is received from the backend) and the standard client (i'm sorry but it's true) will not be able to do that without significant architectural changes.

When NXT (the java part) was initially released it was just one single file with no real architecture to speak of, JL transformed this into the (beautiful imo) architecture we all see today. The standard client is comparable in the way that it consists mainly of big monolithic files where it mixes everything that should not be mixed (talking the Model View Controller parts here). While the client sure looked good it's architecture has always been far from it (although it is approving). Instead focus has been on how it looked instead of it's core, this has been halting progress in the past and it's what we are seeing happening now (and it will happen again in the future).

Quote from: jl777 on June 13, 2015, 06:31:22 am

mark phased orders with a "P" or some other method to indicate they are phased
removing all unconfirmed orders because phased orders are not quite correctly represented as unconfirmed orders is like, well it is not logical.
if phased tx and normal unconfirmed tx are different, then display them a bit differently

For this to work you need to structure the client in a MVC way, where the M (model) part is smart enough to reflect a unified and filtered view of the two different datasources that make up confirmed and unconfirmed transactions. (it are two sources since it requires two API calls - although in NXT+ I've combined most of those into a single call). As you must have noticed before unconfirmed transactions always only appeared at the top of the list, the reason being that raw HTML was simply generated and inserted at the start of the TABLE. That is mixing model and view code (not good), to make things worse the controller code is also mixed into it by adding onclick attributes and such.

To make happen what you propose you need to be able to work on the individual parts of the MVC. Ideally the model provides a unified view of confirmed and unconfirmed and even better is when this data is updated in real time since the model listens for server side events. When done like this an addition like phasing can be supported in the client code by updating a few lines in the model code only.

Quote

MVC example (notice the separation of the three)

Model - https://github.com/fimkrypto/mofowallet/blob/master/app/scripts/providers/activity-provider.js
View -  https://github.com/fimkrypto/mofowallet/blob/master/app/partials/activity.html
Controller - https://github.com/fimkrypto/mofowallet/blob/master/app/scripts/controllers/activity.js

In case you are wondering what the https://github.com/fimkrypto/mofowallet/blob/master/app/scripts/services/entity-provider.js (base class of most models) does.
It basically takes care of all pagination and implements all standard server side events (new block, block popped, ADDED_UNCONFIRMED_TRANSACTION etc..)

Mofowallet does all of the above and is compatible with NXT I had always hoped it was picked up more than it is now.
I sure could use the help and support to make it even better

But it was working perfectly in the last release... As a user all I see is that you took a feature out that was working great for over a year because now it is suddenly too hard? I find this confusing.

This was a very important usability feature and it's replacement is just not user friendly. I feel that it should have been considered more in the design.

it used to be I could do some AE trades in between builds, but now I cant...
the reason is I have to remember every trade I did as opposed to see them pending
so if phased tx not right, then just make the phased tx invisible
to make all tx invisible because phased tx is not right, this is wrong
it is to "throw the baby out with the bath water" because there was a spider floating in the water

And another example: lets say I have 20 orders for 1 asset at different price points. I now wish to cancel some of them using jay/nxtvault. It becomes very easy to lose track of which ones you've cancelled and which you haven't since there is no feedback. On the new screen all you'll see is bid cancellation. No feedback as to which bids.

I want to verify my proposed usage for Phased tx to create an atomic bundle of InstantDEX tx.

I think all I have to do is add a few phasing fields to the existing tx, but that seems too painless.

currently there is a trigger tx that all the other tx use as a referenced tx (it is the InstantDEX fee), so if the fee is paid, all the trade components get confirmed. Except for the rare theoretical edge cases.

By adding:
phased=true
phasingFinishHeight=currentheight+1
phasingVotingModel=4
phasingQuorum=1
phasingLinkedFullHash=feetx_fullhash

instead of using feetx_fullhash as referencedTransactionFullHash

all parties to the bundle would make sure the other parts of the tx are in unconfirmed, but of course in a time sequence to avoid allowing the one with the trigger to get benefit of partial completion. so if Alice is making the feetx, she publishes an unconfirmed tx of her half that references the feetx_fullhash, Bob verifies it before publishing his half. Alice makes sure Bob's tx is there and then publishes the feetx.

Now if we are at block 500000, then the phasingFinishHeight=500001, which means if Alice gets the feetx to the block forger in time, all parts of the bundle are confirmed, if the deadline is missed, then everything expires, but in all cases it is all or nothing and no party is ever in a position to get part of the tx.

If this will work, then it is happening just as fast, but totally atomic. I just realized that in the event that the triggertx is not submitted, then tx fees are incurred for a bundle that is not confirmed...

At the time cost of changing the finish height to +2, could I also make all the Phasing tx also have a normal referencedTransactionFullHash? It would then have both a normal and phased dependency, but this would make it so there is no cost in the cases that the bundle isnt confirmed

James

Technically, the minimum finish height for a phased transaction of type 4 (by-transaction voting) needs to be at least current height plus 2. If you are submitting the phased transaction at blockchain height 50000, it can be included at earliest in block 50001, and finish at 50002. I think this is what you meant anyway.

The risk with such a short finish height is that the phased transaction may fail to be included in block 50001, either unintentionally or due to an evil forger, or an attacker that decides to fill block 50001 with 255 transactions of slightly higher fee to prevent yours from being included. Then it will never be included, because its finish height of 50002 will no longer be acceptable.

The standard way of using by-transaction phasing that I thought of was to allow enough blocks before the finish height to make such attacks not feasible, and also use a deadline limit. The example is: Alice creates a (non-phased) transaction txA that will expire in 6 h, and tells Bob the full hash, but does not broadcast the transaction yet. Bob verifies the transaction data, creates a phased transaction txB that will finish after 800 blocks and uses the full hash of txA as a phasingLinkedFullHash parameter, and broadcasts it. Alice waits until txB is confirmed and has at least 10 confirmations, then broadcasts txA. After that, if txA is included in the blockchain before the txB finish height (and the txA deadline), both will be executed; if txA is prevented from being included in the blockchain (by chance, evil forger, or block filling attack) before the txB finish height, it will expire unconfirmed as its deadline is before the txB finish height (assuming the normal 800 blocks per day average), so both txA and txB will not be executed.

By not waiting for txB to be confirmed in the blockchain, but submitting txA as soon as txB is seen in the unconfirmed pool, there is the risk that txA gets included, but txB does not - again by action of an evil forger, or block filling by third party transactions. The latter attack may work especially if the finish height of txB is very short, because the attacker only needs to fill 2-3 blocks, and then txB will be past its finish height and never included, while txA will be.

If you also make txB reference txA, this will indeed avoid the fee for txB in the case txA is not included in the blockchain before the txB finish height, as txB will wait in the unconfirmed pool for txA to be accepted first. But this is still susceptible to the above evil forger or transaction flooding attacks.

Quote from: jl777 on June 14, 2015, 06:15:14 am

I want to verify my proposed usage for Phased tx to create an atomic bundle of InstantDEX tx.

I think all I have to do is add a few phasing fields to the existing tx, but that seems too painless.

currently there is a trigger tx that all the other tx use as a referenced tx (it is the InstantDEX fee), so if the fee is paid, all the trade components get confirmed. Except for the rare theoretical edge cases.

By adding:
phased=true
phasingFinishHeight=currentheight+1
phasingVotingModel=4
phasingQuorum=1
phasingLinkedFullHash=feetx_fullhash

instead of using feetx_fullhash as referencedTransactionFullHash

all parties to the bundle would make sure the other parts of the tx are in unconfirmed, but of course in a time sequence to avoid allowing the one with the trigger to get benefit of partial completion. so if Alice is making the feetx, she publishes an unconfirmed tx of her half that references the feetx_fullhash, Bob verifies it before publishing his half. Alice makes sure Bob's tx is there and then publishes the feetx.

Now if we are at block 500000, then the phasingFinishHeight=500001, which means if Alice gets the feetx to the block forger in time, all parts of the bundle are confirmed, if the deadline is missed, then everything expires, but in all cases it is all or nothing and no party is ever in a position to get part of the tx.

If this will work, then it is happening just as fast, but totally atomic. I just realized that in the event that the triggertx is not submitted, then tx fees are incurred for a bundle that is not confirmed...

At the time cost of changing the finish height to +2, could I also make all the Phasing tx also have a normal referencedTransactionFullHash? It would then have both a normal and phased dependency, but this would make it so there is no cost in the cases that the bundle isnt confirmed

James

Technically, the minimum finish height for a phased transaction of type 4 (by-transaction voting) needs to be at least current height plus 2. If you are submitting the phased transaction at blockchain height 50000, it can be included at earliest in block 50001, and finish at 50002. I think this is what you meant anyway.

The risk with such a short finish height is that the phased transaction may fail to be included in block 50001, either unintentionally or due to an evil forger, or an attacker that decides to fill block 50001 with 255 transactions of slightly higher fee to prevent yours from being included. Then it will never be included, because its finish height of 50002 will no longer be acceptable.

The standard way of using by-transaction phasing that I thought of was to allow enough blocks before the finish height to make such attacks not feasible, and also use a deadline limit. The example is: Alice creates a (non-phased) transaction txA that will expire in 6 h, and tells Bob the full hash, but does not broadcast the transaction yet. Bob verifies the transaction data, creates a phased transaction txB that will finish after 800 blocks and uses the full hash of txA as a phasingLinkedFullHash parameter, and broadcasts it. Alice waits until txB is confirmed and has at least 10 confirmations, then broadcasts txA. After that, if txA is included in the blockchain before the txB finish height (and the txA deadline), both will be executed; if txA is prevented from being included in the blockchain (by chance, evil forger, or block filling attack) before the txB finish height, it will expire unconfirmed as its deadline is before the txB finish height (assuming the normal 800 blocks per day average), so both txA and txB will not be executed.

By not waiting for txB to be confirmed in the blockchain, but submitting txA as soon as txB is seen in the unconfirmed pool, there is the risk that txA gets included, but txB does not - again by action of an evil forger, or block filling by third party transactions. The latter attack may work especially if the finish height of txB is very short, because the attacker only needs to fill 2-3 blocks, and then txB will be past its finish height and never included, while txA will be.

If you also make txB reference txA, this will indeed avoid the fee for txB in the case txA is not included in the blockchain before the txB finish height, as txB will wait in the unconfirmed pool for txA to be accepted first. But this is still susceptible to the above evil forger or transaction flooding attacks.

if the evaluation logic was to complete a phased tx if all conditions are met, then I can make for longer finish height, but as I understand it a phased tx is only evaluated at the finish height, so if I make it in the far future (yes hours is far future), then it wont finish until then and funds are tied up until then.

so how to make a atomic bundle that resolves one way or another in a few blocks?

what about a phased tx based on revealed secret? both alice an bob can generate a shared secret based seed so either could trigger the tx bundle.

or txB refers to txA and txA refers to feetx? but if the forger can still selectively remove tx, it wont work. so I need to wait until both txA and txB are confirmed (this is a difference over unconfirmed and phased as phased tx will confirm even if not all conditions are met), then and only then will the feetx trigger both as an atomic bundle. of course if we are on a small reorg and only one of them is back in the main chain, we are having an imbalance, but at some point the probabilities become small enough.

if there is no other way to make it safe (only reason to use phased tx), then I will do the following:

1. for small value tx, just use unconfirmed and rely on it being uneconomical for any evil forger to lose a day+ of forging to grab half the trade.

2. for large tx (or at user option) activate phasing where it is like now with the only difference that the feetx (trigger) is not broadcast until there are N confirms for both txA and txB and I will set the finish height to 2*N. where N is on some sliding scale based on total value of tx with max of 10 and min of 3

James

Can you use the new eventRegister and eventWait APIs, and listen to events that are asset transactions, starting from some known state? Currently the events for Trade and AssetTransfer are not accepted, but we can easily add support for those and define other events if needed. This will scale better than trying to retrieve all asset related transactions in chronological order and keep retrieving them (and see if their total count changed, and then see what the new ones are, which if I understand it right is what you are doing now).

Adding an option to request a chronological order for the result of getTrades and getAssetTransfers (instead of the reverse chronological that it is now) is possible, but will be slower than the default (because the database indexes go in the other direction).

Can you also use instead the timestamp parameter to getTrades, to restrict the results only to trades with timestamp newer than the last one you checked? I can also add such timestamp parameter to getAssetTransfer.

Need to also make sure to consider what happens if there is a rollback or switch to a different fork, in which the transactions may have occurred in different order, block timestamps are different, etc.

For getting ask and bid orders too, those are only kept in a table until filled or cancelled, therefore even if there is an API to get them in reverse sort, this sort will not be constant as asks and bids will keep disappearing from the list after being deleted from the table.

And about just getting a list of transaction ids related to an asset id, there is no easy way because there is no asset_id column in the transaction table, this information is inside the attachment bytes, cannot filter by it. Need to get trades and transfers separately, from their respective tables.

Can you use the new eventRegister and eventWait APIs, and listen to events that are asset transactions, starting from some known state? Currently the events for Trade and AssetTransfer are not accepted, but we can easily add support for those and define other events if needed. This will scale better than trying to retrieve all asset related transactions in chronological order and keep retrieving them (and see if their total count changed, and then see what the new ones are, which if I understand it right is what you are doing now).

Adding an option to request a chronological order for the result of getTrades and getAssetTransfers (instead of the reverse chronological that it is now) is possible, but will be slower than the default (because the database indexes go in the other direction).

Can you also use instead the timestamp parameter to getTrades, to restrict the results only to trades with timestamp newer than the last one you checked? I can also add such timestamp parameter to getAssetTransfer.

Need to also make sure to consider what happens if there is a rollback or switch to a different fork, in which the transactions may have occurred in different order, block timestamps are different, etc.

For getting ask and bid orders too, those are only kept in a table until filled or cancelled, therefore even if there is an API to get them in reverse sort, this sort will not be constant as asks and bids will keep disappearing from the list after being deleted from the table.

And about just getting a list of transaction ids related to an asset id, there is no easy way because there is no asset_id column in the transaction table, this information is inside the attachment bytes, cannot filter by it. Need to get trades and transfers separately, from their respective tables.

Thanks. If there is no easy way for this from your standpoint, it certainly makes sense that from the outside using API, it is not really possible.

I think I just need to parse the blockchain. Once its parsed, each new block is the event that will just use the same blockchain parsing

James

c) feetx is confirmed AT the finish height and ? (i think it could confirm both or neither, not sure, probably depends on evaluation order, but if phased tx are evaluated after all the other tx are confirmed should be possible for the referenced tx to confirm on the finish height)

At their finish height, phased transactions are validated and executed before all regular transactions in that block. So if the feetx is in the finish height block, they will not be executed, as feetx will not have been processed yet, it must be in the block before at latest.

P.S. is there ANY way to complete the phased tx as soon as all conditions are met? then I can extend the finish height to much larger amounts

I will revisit this for 1.6, but this compromise was necessary in order to implement by-transaction and by-hash transaction release as just another case of phased transactions. With all the complications in the processing of phased transactions, some of which were discovered very late in the project and I had to do substantial last minute code reorganizations, allowing for variable finish height would have delayed the 1.5 release even more and risk more logic bugs and performance problems (having to check before every block if any of the transactions in it could lead to a phased transaction being finished prematurely, and then adding it to the list of phased transactions finishing in this block, etc).

It was indeed  a big step! very exiting things on the horizon. very much looking forward and invested in this compromise being solved for 1.6.

Cheers! Great work folk.