elective-stereophonic
elective-stereophonic
NRS v1.5.11
singapore
Please login or register.

Login with username, password and session length
Advanced search  

News:

Latest Stable Nxt Client: Nxt 1.12.2

Pages: 1 ... 4 5 [6]  All

Author Topic: NRS v1.5.11  (Read 37607 times)

_mr_e

  • Hero Member
  • *****
  • Karma: +88/-18
  • Offline Offline
  • Posts: 956
    • View Profile
Re: NRS v1.5.11
« Reply #100 on: June 14, 2015, 08:18:00 pm »

So why don't you see your own unconfirmed order getting added to the order book when you place it anymore? I hope this wasn't a purposeful change. It's especially annoying when using jnxt.org/nxt in combination with nxtvault or jay because you no longer have any feedback of your order going through!

Because that's a very difficult thing to do (because of how the data is received from the backend) and the standard client (i'm sorry but it's true) will not be able to do that without significant architectural changes.

When NXT (the java part) was initially released it was just one single file with no real architecture to speak of, JL transformed this into the (beautiful imo) architecture we all see today. The standard client is comparable in the way that it consists mainly of big monolithic files where it mixes everything that should not be mixed (talking the Model View Controller parts here). While the client sure looked good it's architecture has always been far from it (although it is approving). Instead focus has been on how it looked instead of it's core, this has been halting progress in the past and it's what we are seeing happening now (and it will happen again in the future).

mark phased orders with a "P" or some other method to indicate they are phased
removing all unconfirmed orders because phased orders are not quite correctly represented as unconfirmed orders is like, well it is not logical.
if phased tx and normal unconfirmed tx are different, then display them a bit differently

For this to work you need to structure the client in a MVC way, where the M (model) part is smart enough to reflect a unified and filtered view of the two different datasources that make up confirmed and unconfirmed transactions. (it are two sources since it requires two API calls - although in NXT+ I've combined most of those into a single call). As you must have noticed before unconfirmed transactions always only appeared at the top of the list, the reason being that raw HTML was simply generated and inserted at the start of the TABLE. That is mixing model and view code (not good), to make things worse the controller code is also mixed into it by adding onclick attributes and such.

To make happen what you propose you need to be able to work on the individual parts of the MVC. Ideally the model provides a unified view of confirmed and unconfirmed and even better is when this data is updated in real time since the model listens for server side events. When done like this an addition like phasing can be supported in the client code by updating a few lines in the model code only.

Quote
MVC example (notice the separation of the three)

Model - https://github.com/fimkrypto/mofowallet/blob/master/app/scripts/providers/activity-provider.js
View -  https://github.com/fimkrypto/mofowallet/blob/master/app/partials/activity.html
Controller - https://github.com/fimkrypto/mofowallet/blob/master/app/scripts/controllers/activity.js

In case you are wondering what the https://github.com/fimkrypto/mofowallet/blob/master/app/scripts/services/entity-provider.js (base class of most models) does.
It basically takes care of all pagination and implements all standard server side events (new block, block popped, ADDED_UNCONFIRMED_TRANSACTION etc..)

Mofowallet does all of the above and is compatible with NXT I had always hoped it was picked up more than it is now.
I sure could use the help and support to make it even better ;D

But it was working perfectly in the last release... As a user all I see is that you took a feature out that was working great for over a year because now it is suddenly too hard? I find this confusing.

This was a very important usability feature and it's replacement is just not user friendly. I feel that it should have been considered more in the design.
« Last Edit: June 14, 2015, 08:49:18 pm by _mr_e »
Logged

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: NRS v1.5.11
« Reply #101 on: June 14, 2015, 09:04:20 pm »

So why don't you see your own unconfirmed order getting added to the order book when you place it anymore? I hope this wasn't a purposeful change. It's especially annoying when using jnxt.org/nxt in combination with nxtvault or jay because you no longer have any feedback of your order going through!

Because that's a very difficult thing to do (because of how the data is received from the backend) and the standard client (i'm sorry but it's true) will not be able to do that without significant architectural changes.

When NXT (the java part) was initially released it was just one single file with no real architecture to speak of, JL transformed this into the (beautiful imo) architecture we all see today. The standard client is comparable in the way that it consists mainly of big monolithic files where it mixes everything that should not be mixed (talking the Model View Controller parts here). While the client sure looked good it's architecture has always been far from it (although it is approving). Instead focus has been on how it looked instead of it's core, this has been halting progress in the past and it's what we are seeing happening now (and it will happen again in the future).

mark phased orders with a "P" or some other method to indicate they are phased
removing all unconfirmed orders because phased orders are not quite correctly represented as unconfirmed orders is like, well it is not logical.
if phased tx and normal unconfirmed tx are different, then display them a bit differently

For this to work you need to structure the client in a MVC way, where the M (model) part is smart enough to reflect a unified and filtered view of the two different datasources that make up confirmed and unconfirmed transactions. (it are two sources since it requires two API calls - although in NXT+ I've combined most of those into a single call). As you must have noticed before unconfirmed transactions always only appeared at the top of the list, the reason being that raw HTML was simply generated and inserted at the start of the TABLE. That is mixing model and view code (not good), to make things worse the controller code is also mixed into it by adding onclick attributes and such.

To make happen what you propose you need to be able to work on the individual parts of the MVC. Ideally the model provides a unified view of confirmed and unconfirmed and even better is when this data is updated in real time since the model listens for server side events. When done like this an addition like phasing can be supported in the client code by updating a few lines in the model code only.

Quote
MVC example (notice the separation of the three)

Model - https://github.com/fimkrypto/mofowallet/blob/master/app/scripts/providers/activity-provider.js
View -  https://github.com/fimkrypto/mofowallet/blob/master/app/partials/activity.html
Controller - https://github.com/fimkrypto/mofowallet/blob/master/app/scripts/controllers/activity.js

In case you are wondering what the https://github.com/fimkrypto/mofowallet/blob/master/app/scripts/services/entity-provider.js (base class of most models) does.
It basically takes care of all pagination and implements all standard server side events (new block, block popped, ADDED_UNCONFIRMED_TRANSACTION etc..)

Mofowallet does all of the above and is compatible with NXT I had always hoped it was picked up more than it is now.
I sure could use the help and support to make it even better ;D

But it was working perfectly in the last release... As a user all I see is that you took a feature out that was working great for over a year because now it is suddenly too hard? I find this confusing.

This was a very important usability feature and it's replacement is just not user friendly. I feel that it should have been considered more in the design.
it used to be I could do some AE trades in between builds, but now I cant...
the reason is I have to remember every trade I did as opposed to see them pending
so if phased tx not right, then just make the phased tx invisible
to make all tx invisible because phased tx is not right, this is wrong
it is to "throw the baby out with the bath water" because there was a spider floating in the water
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

_mr_e

  • Hero Member
  • *****
  • Karma: +88/-18
  • Offline Offline
  • Posts: 956
    • View Profile
Re: NRS v1.5.11
« Reply #102 on: June 14, 2015, 09:14:29 pm »

So why don't you see your own unconfirmed order getting added to the order book when you place it anymore? I hope this wasn't a purposeful change. It's especially annoying when using jnxt.org/nxt in combination with nxtvault or jay because you no longer have any feedback of your order going through!

Because that's a very difficult thing to do (because of how the data is received from the backend) and the standard client (i'm sorry but it's true) will not be able to do that without significant architectural changes.

When NXT (the java part) was initially released it was just one single file with no real architecture to speak of, JL transformed this into the (beautiful imo) architecture we all see today. The standard client is comparable in the way that it consists mainly of big monolithic files where it mixes everything that should not be mixed (talking the Model View Controller parts here). While the client sure looked good it's architecture has always been far from it (although it is approving). Instead focus has been on how it looked instead of it's core, this has been halting progress in the past and it's what we are seeing happening now (and it will happen again in the future).

mark phased orders with a "P" or some other method to indicate they are phased
removing all unconfirmed orders because phased orders are not quite correctly represented as unconfirmed orders is like, well it is not logical.
if phased tx and normal unconfirmed tx are different, then display them a bit differently

For this to work you need to structure the client in a MVC way, where the M (model) part is smart enough to reflect a unified and filtered view of the two different datasources that make up confirmed and unconfirmed transactions. (it are two sources since it requires two API calls - although in NXT+ I've combined most of those into a single call). As you must have noticed before unconfirmed transactions always only appeared at the top of the list, the reason being that raw HTML was simply generated and inserted at the start of the TABLE. That is mixing model and view code (not good), to make things worse the controller code is also mixed into it by adding onclick attributes and such.

To make happen what you propose you need to be able to work on the individual parts of the MVC. Ideally the model provides a unified view of confirmed and unconfirmed and even better is when this data is updated in real time since the model listens for server side events. When done like this an addition like phasing can be supported in the client code by updating a few lines in the model code only.

Quote
MVC example (notice the separation of the three)

Model - https://github.com/fimkrypto/mofowallet/blob/master/app/scripts/providers/activity-provider.js
View -  https://github.com/fimkrypto/mofowallet/blob/master/app/partials/activity.html
Controller - https://github.com/fimkrypto/mofowallet/blob/master/app/scripts/controllers/activity.js

In case you are wondering what the https://github.com/fimkrypto/mofowallet/blob/master/app/scripts/services/entity-provider.js (base class of most models) does.
It basically takes care of all pagination and implements all standard server side events (new block, block popped, ADDED_UNCONFIRMED_TRANSACTION etc..)

Mofowallet does all of the above and is compatible with NXT I had always hoped it was picked up more than it is now.
I sure could use the help and support to make it even better ;D

But it was working perfectly in the last release... As a user all I see is that you took a feature out that was working great for over a year because now it is suddenly too hard? I find this confusing.

This was a very important usability feature and it's replacement is just not user friendly. I feel that it should have been considered more in the design.
it used to be I could do some AE trades in between builds, but now I cant...
the reason is I have to remember every trade I did as opposed to see them pending
so if phased tx not right, then just make the phased tx invisible
to make all tx invisible because phased tx is not right, this is wrong
it is to "throw the baby out with the bath water" because there was a spider floating in the water
And another example: lets say I have 20 orders for 1 asset at different price points. I now wish to cancel some of them using jay/nxtvault. It becomes very easy to lose track of which ones you've cancelled and which you haven't since there is no feedback. On the new screen all you'll see is bid cancellation. No feedback as to which bids.
Logged

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: NRS v1.5.11
« Reply #103 on: June 15, 2015, 05:19:32 am »

I need to get a historical list of all asset transactions, preferably on a per asset basis.

all I could find are getAllOpenAskOrders and getAllOpenBidOrders, but this is a dynamic list and only for open orders.

getAskOrders and getBidOrders are sorted by price, so I cant efficiently detect changes.

At this point it looks like I have to parse the entire blockchain to be able to do this, but that seems so 2014.

Hopefully I missed the right API call

James

P.S. I see Transaction.ADDED_CONFIRMED_TRANSACTIONS as one of the events, but this requires that I have a current state of all AE transactions, which requires scanning all accounts or all blocks as near as I can tell
« Last Edit: June 15, 2015, 05:21:47 am by jl777 »
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

Jean-Luc

  • Core Dev
  • Hero Member
  • *****
  • Karma: +816/-81
  • Offline Offline
  • Posts: 1610
    • View Profile
Re: NRS v1.5.11
« Reply #104 on: June 15, 2015, 10:30:55 pm »

I want to verify my proposed usage for Phased tx to create an atomic bundle of InstantDEX tx.

I think all I have to do is add a few phasing fields to the existing tx, but that seems too painless.

currently there is a trigger tx that all the other tx use as a referenced tx (it is the InstantDEX fee), so if the fee is paid, all the trade components get confirmed. Except for the rare theoretical edge cases.

By adding:
phased=true
phasingFinishHeight=currentheight+1
phasingVotingModel=4
phasingQuorum=1
phasingLinkedFullHash=feetx_fullhash

instead of using feetx_fullhash as referencedTransactionFullHash

all parties to the bundle would make sure the other parts of the tx are in unconfirmed, but of course in a time sequence to avoid allowing the one with the trigger to get benefit of partial completion. so if Alice is making the feetx, she publishes an unconfirmed tx of her half that references the feetx_fullhash, Bob verifies it before publishing his half. Alice makes sure Bob's tx is there and then publishes the feetx.

Now if we are at block 500000, then the phasingFinishHeight=500001, which means if Alice gets the feetx to the block forger in time, all parts of the bundle are confirmed, if the deadline is missed, then everything expires, but in all cases it is all or nothing and no party is ever in a position to get part of the tx.

If this will work, then it is happening just as fast, but totally atomic. I just realized that in the event that the triggertx is not submitted, then tx fees are incurred for a bundle that is not confirmed...

At the time cost of changing the finish height to +2, could I also make all the Phasing tx also have a normal referencedTransactionFullHash? It would then have both a normal and phased dependency, but this would make it so there is no cost in the cases that the bundle isnt confirmed

James

Technically, the minimum finish height for a phased transaction of type 4 (by-transaction voting) needs to be at least current height plus 2. If you are submitting the phased transaction at blockchain height 50000, it can be included at earliest in block 50001, and finish at 50002. I think this is what you meant anyway.

The risk with such a short finish height is that the phased transaction may fail to be included in block 50001, either unintentionally or due to an evil forger, or an attacker that decides to fill block 50001 with 255 transactions of slightly higher fee to prevent yours from being included. Then it will never be included, because its finish height of 50002 will no longer be acceptable.

The standard way of using by-transaction phasing that I thought of was to allow enough blocks before the finish height to make such attacks not feasible, and also use a deadline limit. The example is: Alice creates a (non-phased) transaction txA that will expire in 6 h, and tells Bob the full hash, but does not broadcast the transaction yet. Bob verifies the transaction data, creates a phased transaction txB that will finish after 800 blocks and uses the full hash of txA as a phasingLinkedFullHash parameter, and broadcasts it. Alice waits until txB is confirmed and has at least 10 confirmations, then broadcasts txA. After that, if txA is included in the blockchain before the txB finish height (and the txA deadline), both will be executed; if txA is prevented from being included in the blockchain (by chance, evil forger, or block filling attack) before the txB finish height, it will expire unconfirmed as its deadline is before the txB finish height (assuming the normal 800 blocks per day average), so both txA and txB will not be executed.

By not waiting for txB to be confirmed in the blockchain, but submitting txA as soon as txB is seen in the unconfirmed pool, there is the risk that txA gets included, but txB does not - again by action of an evil forger, or block filling by third party transactions. The latter attack may work especially if the finish height of txB is very short, because the attacker only needs to fill 2-3 blocks, and then txB will be past its finish height and never included, while txA will be.

If you also make txB reference txA, this will indeed avoid the fee for txB in the case txA is not included in the blockchain before the txB finish height, as txB will wait in the unconfirmed pool for txA to be accepted first. But this is still susceptible to the above evil forger or transaction flooding attacks.


Logged
GPG key fingerprint: 263A 9EB0 29CF C77A 3D06  FD13 811D 6940 E1E4 240C
NXT-X4LF-9A4G-WN9Z-2R322

Jean-Luc

  • Core Dev
  • Hero Member
  • *****
  • Karma: +816/-81
  • Offline Offline
  • Posts: 1610
    • View Profile
Re: NRS v1.5.11
« Reply #105 on: June 15, 2015, 10:39:41 pm »

I need to get a historical list of all asset transactions, preferably on a per asset basis.
Is getTrades or getAssetTransfers not enough? If you need the actual orders, not only the completed trades, those are indeed not kept after they are filled (or cancelled). Although you can still use getTrades and get the ask/bid order ids for each trade, which are the transaction ids of the orders, except probably not sorted the way you want.
Logged
GPG key fingerprint: 263A 9EB0 29CF C77A 3D06  FD13 811D 6940 E1E4 240C
NXT-X4LF-9A4G-WN9Z-2R322

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: NRS v1.5.11
« Reply #106 on: June 15, 2015, 10:55:16 pm »

I want to verify my proposed usage for Phased tx to create an atomic bundle of InstantDEX tx.

I think all I have to do is add a few phasing fields to the existing tx, but that seems too painless.

currently there is a trigger tx that all the other tx use as a referenced tx (it is the InstantDEX fee), so if the fee is paid, all the trade components get confirmed. Except for the rare theoretical edge cases.

By adding:
phased=true
phasingFinishHeight=currentheight+1
phasingVotingModel=4
phasingQuorum=1
phasingLinkedFullHash=feetx_fullhash

instead of using feetx_fullhash as referencedTransactionFullHash

all parties to the bundle would make sure the other parts of the tx are in unconfirmed, but of course in a time sequence to avoid allowing the one with the trigger to get benefit of partial completion. so if Alice is making the feetx, she publishes an unconfirmed tx of her half that references the feetx_fullhash, Bob verifies it before publishing his half. Alice makes sure Bob's tx is there and then publishes the feetx.

Now if we are at block 500000, then the phasingFinishHeight=500001, which means if Alice gets the feetx to the block forger in time, all parts of the bundle are confirmed, if the deadline is missed, then everything expires, but in all cases it is all or nothing and no party is ever in a position to get part of the tx.

If this will work, then it is happening just as fast, but totally atomic. I just realized that in the event that the triggertx is not submitted, then tx fees are incurred for a bundle that is not confirmed...

At the time cost of changing the finish height to +2, could I also make all the Phasing tx also have a normal referencedTransactionFullHash? It would then have both a normal and phased dependency, but this would make it so there is no cost in the cases that the bundle isnt confirmed

James

Technically, the minimum finish height for a phased transaction of type 4 (by-transaction voting) needs to be at least current height plus 2. If you are submitting the phased transaction at blockchain height 50000, it can be included at earliest in block 50001, and finish at 50002. I think this is what you meant anyway.

The risk with such a short finish height is that the phased transaction may fail to be included in block 50001, either unintentionally or due to an evil forger, or an attacker that decides to fill block 50001 with 255 transactions of slightly higher fee to prevent yours from being included. Then it will never be included, because its finish height of 50002 will no longer be acceptable.

The standard way of using by-transaction phasing that I thought of was to allow enough blocks before the finish height to make such attacks not feasible, and also use a deadline limit. The example is: Alice creates a (non-phased) transaction txA that will expire in 6 h, and tells Bob the full hash, but does not broadcast the transaction yet. Bob verifies the transaction data, creates a phased transaction txB that will finish after 800 blocks and uses the full hash of txA as a phasingLinkedFullHash parameter, and broadcasts it. Alice waits until txB is confirmed and has at least 10 confirmations, then broadcasts txA. After that, if txA is included in the blockchain before the txB finish height (and the txA deadline), both will be executed; if txA is prevented from being included in the blockchain (by chance, evil forger, or block filling attack) before the txB finish height, it will expire unconfirmed as its deadline is before the txB finish height (assuming the normal 800 blocks per day average), so both txA and txB will not be executed.

By not waiting for txB to be confirmed in the blockchain, but submitting txA as soon as txB is seen in the unconfirmed pool, there is the risk that txA gets included, but txB does not - again by action of an evil forger, or block filling by third party transactions. The latter attack may work especially if the finish height of txB is very short, because the attacker only needs to fill 2-3 blocks, and then txB will be past its finish height and never included, while txA will be.

If you also make txB reference txA, this will indeed avoid the fee for txB in the case txA is not included in the blockchain before the txB finish height, as txB will wait in the unconfirmed pool for txA to be accepted first. But this is still susceptible to the above evil forger or transaction flooding attacks.
if the evaluation logic was to complete a phased tx if all conditions are met, then I can make for longer finish height, but as I understand it a phased tx is only evaluated at the finish height, so if I make it in the far future (yes hours is far future), then it wont finish until then and funds are tied up until then.

so how to make a atomic bundle that resolves one way or another in a few blocks?

what about a phased tx based on revealed secret? both alice an bob can generate a shared secret based seed so either could trigger the tx bundle.

or txB refers to txA and txA refers to feetx? but if the forger can still selectively remove tx, it wont work. so I need to wait until both txA and txB are confirmed (this is a difference over unconfirmed and phased as phased tx will confirm even if not all conditions are met), then and only then will the feetx trigger both as an atomic bundle. of course if we are on a small reorg and only one of them is back in the main chain, we are having an imbalance, but at some point the probabilities become small enough.

if there is no other way to make it safe (only reason to use phased tx), then I will do the following:

1. for small value tx, just use unconfirmed and rely on it being uneconomical for any evil forger to lose a day+ of forging to grab half the trade.

2. for large tx (or at user option) activate phasing where it is like now with the only difference that the feetx (trigger) is not broadcast until there are N confirms for both txA and txB and I will set the finish height to 2*N. where N is on some sliding scale based on total value of tx with max of 10 and min of 3

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: NRS v1.5.11
« Reply #107 on: June 15, 2015, 11:07:15 pm »

I need to get a historical list of all asset transactions, preferably on a per asset basis.
Is getTrades or getAssetTransfers not enough? If you need the actual orders, not only the completed trades, those are indeed not kept after they are filled (or cancelled). Although you can still use getTrades and get the ask/bid order ids for each trade, which are the transaction ids of the orders, except probably not sorted the way you want.
for getTrades and getAssetTransfers it works fine as there is a list of tx sorted by time, but i do need to iterate through all 500+ assets. But what about the bids and asks? The reason this is needed is that I have the following logic:

1. if init, load entire list
2. if not, then iterate from 0 to N until an existing tx is found and then add from there back to index 0

what i need is a txlist with an increasing index. it is very problematic to have a tx that is index 0, until it becomes index 1, until it becomes index 2, etc. I need a static txid <-> index mapping

now the allopentrades is the only one that is reverse chronological and I can reverse that using the above logic, except of course when any open trade is closed, all the indices change.

without being able to map a txid to a static index, then i need to build one from scratch, which means rescanning the blockchain like I did last year.

Maybe it is easy for you to add a chronological getAssetTransaction? That would be ideal, as that is what my code needs to update its internal state in an event driven manner. If I can find out how many transactions assetid (with or without account) has, then I can iterate from 0 to there. Then to generate an event, I iterate through assets and find the number of total transactions, if more than last time, iterate from the last time to the current number as each of these are the new events that are not seen before.

As you can see to recreate this chronological by asset list is quite a chore with the way the current API is returning. I realize it is most important to provide a sorted orderbook, but when the txindex is not a constant, I cant use it as a reference, so that means I need to requery the DB each time.

this wouldnt be a problem, expect that the InstantDEX orderbook is propagating in tens of milliseconds, so even a couple HDD accesses from the DB is taking more time than sending the order across the entire network

James

P.S. even if it is possible to generate events from the current point forward, without being able to initialize state to the current point, I would need to resort to querying 1000 assets, bids, asks, trades, or just rescan the blockchain.
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

Jean-Luc

  • Core Dev
  • Hero Member
  • *****
  • Karma: +816/-81
  • Offline Offline
  • Posts: 1610
    • View Profile
Re: NRS v1.5.11
« Reply #108 on: June 17, 2015, 09:13:58 pm »

Can you use the new eventRegister and eventWait APIs, and listen to events that are asset transactions, starting from some known state? Currently the events for Trade and AssetTransfer are not accepted, but we can easily add support for those and define other events if needed. This will scale better than trying to retrieve all asset related transactions in chronological order and keep retrieving them (and see if their total count changed, and then see what the new ones are, which if I understand it right is what you are doing now).

Adding an option to request a chronological order for the result of getTrades and getAssetTransfers (instead of the reverse chronological that it is now) is possible, but will be slower than the default (because the database indexes go in the other direction).

Can you also use instead the timestamp parameter to getTrades, to restrict the results only to trades with timestamp newer than the last one you checked? I can also add such timestamp parameter to getAssetTransfer.

Need to also make sure to consider what happens if there is a rollback or switch to a different fork, in which the transactions may have occurred in different order, block timestamps are different, etc.

For getting ask and bid orders too, those are only kept in a table until filled or cancelled, therefore even if there is an API to get them in reverse sort, this sort will not be constant as asks and bids will keep disappearing from the list after being deleted from the table.

And about just getting a list of transaction ids related to an asset id, there is no easy way because there is no asset_id column in the transaction table, this information is inside the attachment bytes, cannot filter by it. Need to get trades and transfers separately, from their respective tables.
Logged
GPG key fingerprint: 263A 9EB0 29CF C77A 3D06  FD13 811D 6940 E1E4 240C
NXT-X4LF-9A4G-WN9Z-2R322

Jean-Luc

  • Core Dev
  • Hero Member
  • *****
  • Karma: +816/-81
  • Offline Offline
  • Posts: 1610
    • View Profile
Re: NRS v1.5.11
« Reply #109 on: June 17, 2015, 09:28:59 pm »

what about a phased tx based on revealed secret? both alice an bob can generate a shared secret based seed so either could trigger the tx bundle.
Reveal secret (by-hash phasing) is very similar indeed to the by-transaction, if you imagine the transaction itself being the secret (more precisely, its signature, which should not be revealed to the other party until ready). Using reveal secret requires three transactions though instead of just two, txA, txB, and the third transaction that reveals the secret (which anyone can send).

Quote
this is a difference over unconfirmed and phased as phased tx will confirm even if not all conditions are met
This is really the fundamental difference in using referenced transactions or phased transactions. With referenced only, they have to stay in the unconfirmed, you can't avoid evil forger or blockchain filling attacks. With phased, they get confirmed, and if you wait for sufficient number of confirmations, it is guaranteed that they will stay in the blockchain and the above attacks are avoided.

Quote
2. for large tx (or at user option) activate phasing where it is like now with the only difference that the feetx (trigger) is not broadcast until there are N confirms for both txA and txB and I will set the finish height to 2*N. where N is on some sliding scale based on total value of tx with max of 10 and min of 3
Make sure to set them to finish at the same height, to avoid the edge case of the trigger transaction getting included after only one of the phased transactions has finished (and then releasing the remaining one only). And set the deadline for the release transaction to a time which should be before the finish height (this unfortunately involves making a guess how many blocks per minute we get on average), to be certain it will have expired if someone tries to delay its acceptance until after the finish.
Logged
GPG key fingerprint: 263A 9EB0 29CF C77A 3D06  FD13 811D 6940 E1E4 240C
NXT-X4LF-9A4G-WN9Z-2R322

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: NRS v1.5.11
« Reply #110 on: June 18, 2015, 12:32:28 am »

Can you use the new eventRegister and eventWait APIs, and listen to events that are asset transactions, starting from some known state? Currently the events for Trade and AssetTransfer are not accepted, but we can easily add support for those and define other events if needed. This will scale better than trying to retrieve all asset related transactions in chronological order and keep retrieving them (and see if their total count changed, and then see what the new ones are, which if I understand it right is what you are doing now).

Adding an option to request a chronological order for the result of getTrades and getAssetTransfers (instead of the reverse chronological that it is now) is possible, but will be slower than the default (because the database indexes go in the other direction).

Can you also use instead the timestamp parameter to getTrades, to restrict the results only to trades with timestamp newer than the last one you checked? I can also add such timestamp parameter to getAssetTransfer.

Need to also make sure to consider what happens if there is a rollback or switch to a different fork, in which the transactions may have occurred in different order, block timestamps are different, etc.

For getting ask and bid orders too, those are only kept in a table until filled or cancelled, therefore even if there is an API to get them in reverse sort, this sort will not be constant as asks and bids will keep disappearing from the list after being deleted from the table.

And about just getting a list of transaction ids related to an asset id, there is no easy way because there is no asset_id column in the transaction table, this information is inside the attachment bytes, cannot filter by it. Need to get trades and transfers separately, from their respective tables.
Thanks. If there is no easy way for this from your standpoint, it certainly makes sense that from the outside using API, it is not really possible.

I think I just need to parse the blockchain. Once its parsed, each new block is the event that will just use the same blockchain parsing

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: NRS v1.5.11
« Reply #111 on: June 18, 2015, 12:40:47 am »

what about a phased tx based on revealed secret? both alice an bob can generate a shared secret based seed so either could trigger the tx bundle.
Reveal secret (by-hash phasing) is very similar indeed to the by-transaction, if you imagine the transaction itself being the secret (more precisely, its signature, which should not be revealed to the other party until ready). Using reveal secret requires three transactions though instead of just two, txA, txB, and the third transaction that reveals the secret (which anyone can send).

Quote
this is a difference over unconfirmed and phased as phased tx will confirm even if not all conditions are met
This is really the fundamental difference in using referenced transactions or phased transactions. With referenced only, they have to stay in the unconfirmed, you can't avoid evil forger or blockchain filling attacks. With phased, they get confirmed, and if you wait for sufficient number of confirmations, it is guaranteed that they will stay in the blockchain and the above attacks are avoided.

Quote
2. for large tx (or at user option) activate phasing where it is like now with the only difference that the feetx (trigger) is not broadcast until there are N confirms for both txA and txB and I will set the finish height to 2*N. where N is on some sliding scale based on total value of tx with max of 10 and min of 3
Make sure to set them to finish at the same height, to avoid the edge case of the trigger transaction getting included after only one of the phased transactions has finished (and then releasing the remaining one only). And set the deadline for the release transaction to a time which should be before the finish height (this unfortunately involves making a guess how many blocks per minute we get on average), to be certain it will have expired if someone tries to delay its acceptance until after the finish.
I think I am understanding the issues better. My compromise is to set the finish height at currentblock+7 for both the "money" tx, ie txA and txB. then the feetx (trigger) is both used as normal referencedtx and the phased reference and a normal timeout of an hour or two.

I think as long as both txA and txB are at the same height, the following are the possiblities:
a) feetx is confirmed before the finish height and both txA and txB happen
b) feetx is not confirmed before the finish height and neither confirm.
c) feetx is confirmed AT the finish height and ? (i think it could confirm both or neither, not sure, probably depends on evaluation order, but if phased tx are evaluated after all the other tx are confirmed should be possible for the referenced tx to confirm on the finish height)

If the feetx is released as soon as both txA and txB are in the unconfirmed, then the attacker would have to control the next 6 blocks in order to have a successful attack. Considering all the things that have to be done and especially that a trade needs to be pending with the attacker specifically, I think this 6 block requirement makes it impractical.

It seems that unless an attacker is forging 6 blocks in a row, the max possible economic loss is the feetx.

Did I miss an edge case?

James

P.S. is there ANY way to complete the phased tx as soon as all conditions are met? then I can extend the finish height to much larger amounts
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

Jean-Luc

  • Core Dev
  • Hero Member
  • *****
  • Karma: +816/-81
  • Offline Offline
  • Posts: 1610
    • View Profile
Re: NRS v1.5.11
« Reply #112 on: June 18, 2015, 10:52:45 am »

c) feetx is confirmed AT the finish height and ? (i think it could confirm both or neither, not sure, probably depends on evaluation order, but if phased tx are evaluated after all the other tx are confirmed should be possible for the referenced tx to confirm on the finish height)
At their finish height, phased transactions are validated and executed before all regular transactions in that block. So if the feetx is in the finish height block, they will not be executed, as feetx will not have been processed yet, it must be in the block before at latest.

Quote
P.S. is there ANY way to complete the phased tx as soon as all conditions are met? then I can extend the finish height to much larger amounts
I will revisit this for 1.6, but this compromise was necessary in order to implement by-transaction and by-hash transaction release as just another case of phased transactions. With all the complications in the processing of phased transactions, some of which were discovered very late in the project and I had to do substantial last minute code reorganizations, allowing for variable finish height would have delayed the 1.5 release even more and risk more logic bugs and performance problems (having to check before every block if any of the transactions in it could lead to a phased transaction being finished prematurely, and then adding it to the list of phased transactions finishing in this block, etc).
Logged
GPG key fingerprint: 263A 9EB0 29CF C77A 3D06  FD13 811D 6940 E1E4 240C
NXT-X4LF-9A4G-WN9Z-2R322

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: NRS v1.5.11
« Reply #113 on: June 18, 2015, 11:22:21 am »

c) feetx is confirmed AT the finish height and ? (i think it could confirm both or neither, not sure, probably depends on evaluation order, but if phased tx are evaluated after all the other tx are confirmed should be possible for the referenced tx to confirm on the finish height)
At their finish height, phased transactions are validated and executed before all regular transactions in that block. So if the feetx is in the finish height block, they will not be executed, as feetx will not have been processed yet, it must be in the block before at latest.

Quote
P.S. is there ANY way to complete the phased tx as soon as all conditions are met? then I can extend the finish height to much larger amounts
I will revisit this for 1.6, but this compromise was necessary in order to implement by-transaction and by-hash transaction release as just another case of phased transactions. With all the complications in the processing of phased transactions, some of which were discovered very late in the project and I had to do substantial last minute code reorganizations, allowing for variable finish height would have delayed the 1.5 release even more and risk more logic bugs and performance problems (having to check before every block if any of the transactions in it could lead to a phased transaction being finished prematurely, and then adding it to the list of phased transactions finishing in this block, etc).
Makes sense. One step at a time, and it was a big step!

It would be really important to make things as fast as possible, without losing security. I know it is ugly, but if it is possible to get just one variant to be able to do early completion, then that is all I need, especially if it is the by-transaction variant.

The same block would be a nice improvement, but doesnt make sense until after adaptive finish height is working. It would be worth a hardfork to get this.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

JGalt

  • Jr. Member
  • **
  • Karma: +18/-0
  • Offline Offline
  • Posts: 97
    • View Profile
Re: NRS v1.5.11
« Reply #114 on: June 19, 2015, 07:32:04 pm »

It was indeed  a big step! very exiting things on the horizon. very much looking forward and invested in this compromise being solved for 1.6.

Cheers! Great work folk.
Logged
Twitter @juansgalt
Net 2.0 journalist, privacy zealot, wannebe entrepreneur.
Pages: 1 ... 4 5 [6]  All
 

elective-stereophonic
elective-stereophonic
assembly
assembly