elective-stereophonic
elective-stereophonic
Multigateway status reports singapore
Please login or register.

Login with username, password and session length
Advanced search  

News:

Latest Stable Nxt Client: Nxt 1.12.2

Pages: 1 ... 6 7 [8] 9 10 11  All

Author Topic: Multigateway status reports  (Read 39659 times)

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #140 on: May 13, 2014, 10:31:04 am »

after 4hrs sleep after all nighter due to AE launch I wasnt able to code today,but I did make a breakthrough on the InstantDEX design. now even if evilbob forges the next block and is doing a direct tx with you, the worst he can do is not do the trade he agreed to. considering how rare it will be and it is just an annoyance without any funds lost, I consider this attack vector dealt with!

Have a few pretty major things cooking, but not ready to announce anything yet. very exciting times for NXT!!!

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

Daedelus

  • Hero Member
  • *****
  • Karma: +230/-12
  • Offline Offline
  • Posts: 3280
    • View Profile
Re: Multigateway status reports
« Reply #141 on: May 13, 2014, 10:53:27 am »

Are these announcements of things brand new porjects or have you trailed them a little already?  ;D
Logged
NXT: NXT-4CS7-S4N5-PTH5-A8R2Q

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #142 on: May 13, 2014, 11:05:24 am »

Are these announcements of things brand new porjects or have you trailed them a little already?  ;D
I am working on three deals. One tech, one finance, and one that is neither. all relate to NXT of course
This is what happens when I dont get enough sleep. I cant code, but I have to do something :)

Now that things have calmed down I will be going back into coding mode. I had been providing personal inventory for both InstantDEX and NXTcoinsco, but after what little is still on the books are gone, it will be up to the early purchasers to provide some inventory. I have found that I cant code and trade at the same time, at least not effectively as it seems to use very similar parts of my brain.

James

P.S. I have put small buy walls to provide a trading range for traders. Those I dont have to think about so they will stay.
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

swartzfeger

  • Hero Member
  • *****
  • Karma: +50/-1
  • Offline Offline
  • Posts: 611
  • I bent my wookie
    • View Profile
    • https://www.instagram.com/swartzfeger/
Re: Multigateway status reports
« Reply #143 on: May 14, 2014, 11:35:28 pm »

James, I've been remiss in testing because my smarter partner (chanc3r) has been unavailable due to work. Is there anything I can help test, break, etc?
Logged

CoinTropololis_JustaBit

  • Hero Member
  • *****
  • Karma: +144/-11
  • Offline Offline
  • Posts: 727
    • View Profile
Re: Multigateway status reports
« Reply #144 on: May 14, 2014, 11:47:54 pm »

Exciting times indeed. ;)

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #145 on: May 15, 2014, 01:20:43 am »

James, I've been remiss in testing because my smarter partner (chanc3r) has been unavailable due to work. Is there anything I can help test, break, etc?
I really would like to see people other than me prove that they can chat directly
so just install v03 and establish chat with anybody else that would help
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #146 on: May 16, 2014, 06:37:42 am »

You might have noticed I have been doing a lot of financing recently. The reason for this is that the bitcoind via libcurl is just sooooo slow. It take 20 minutes to an hour to reload and everyday it gets bigger and slower.
With NXT, I can reprocess the whole blockchain in a few minutes. So during this eternal wait, I check forum, PM's, an idea forms and next thing you know there is a new asset.

I think I need to speed up this MGW initialization. It shouldnt be restarted that often once it is deployed but it just isnt practical to debug now. To date, all MGW state info was in the blockchains. this is nice and decentralized but requires a full blockchain scan each time. Too slow.

so, I will code up a way to massively speed this part up to allow rapid cycle debugging. This will take a bit of time. More when I know how much time.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #147 on: May 18, 2014, 01:19:14 am »

I have proof libcurl is the devil!
I tried all sorts of arbitrary values to delay between calls to prevent from going mysteriously non-responsive for many seconds.

Of course when i used 666 milliseconds, the errors go away.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

psic4t

  • Jr. Member
  • **
  • Karma: +4/-0
  • Offline Offline
  • Posts: 55
  • NXT-NHPU-6WBD-CPNG-2CXCE
    • View Profile
Re: Multigateway status reports
« Reply #148 on: May 18, 2014, 01:39:58 am »

Of course when i used 666 milliseconds, the errors go away.

Hahaha. Beat it!
Logged

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #149 on: May 21, 2014, 08:34:22 pm »

Finally, now that AE launch is a week behind us, I able to finish all the nontech work with more than a few hours left in the day!

That means I can actually write code again.

So back to the MGW issue that I have to put silly pauses before each RPC command to reduce the number of 10 seconds sleeps that seems to be the only way to get things working again. OK this doesnt sound that bad until you count the number of RPC calls are needed to rescan the BTC blockchain. Ever wonder why reprocessing the blockchain takes so long?

The reason is that it does not seem to be optimized at all. They assumed that the network will be the bottleneck and did not count on tens of thousands of requests per second that my code issues. I use hashtables that have on average two iterations regardless of table entries so it is effectively constant time searching. Basically everything is really fast, except for the libcurl.

Each BTC block has a lot of transactions, so reprocessing requires 1 RPC to get the block and N*2 RPC for each tx in the block since I have to getrawtx and decode it. bitcoind does not properly associate multisig addrs to accts. It really has been a challenge to make MGW work properly

Anyway the practical problem was that each time I made a new version I had to wait for 30+ minutes to reprocess. To me that is a LONG time. So I check PM's, catchup on forum and next thing you know I am far away from debugging. By the time I notice that it has finally reprocessed I usually cant remember what change I just made.

Something had to be done. I like interactive debugging. Make a small change, verify it works, repeat until all bugs are fixed. I make hundreds of test versions per day when actively debugging.

Anyway, I was considering gutting the current control flow to deal with this issue, but I just couldnt get myself to do it. Afterall, it is 99% working and any new implementation has no guarantee of being that much better. But to speed it up requires a massive overall. Or does it?

Dont know why I didnt think of it before, lack of sleep and time is what I blame :)
Good news is that I am current waiting for the test of the sped up version. Still not great, but instead of 30+ minutes, less than 5 minutes to reprocess!

So, I can finally semi-interactively debug MGW

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #150 on: May 22, 2014, 12:08:08 am »

Must faster now, processing around 100000 requests per second. Still not optimally fast, but certainly much better than before, about 100 times faster!

Now I can run all four daemons and get a reasonable debug cycle, but it will take an hour or so to initialize the first time.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

ebby

  • Full Member
  • ***
  • Karma: +21/-2
  • Offline Offline
  • Posts: 190
    • View Profile
Re: Multigateway status reports
« Reply #151 on: May 23, 2014, 05:14:52 pm »

Nice to see some success again with MGW, thanks jl777!

Any eta when MGW will be ready for Mainnet? I think everyone is waiting for it.
Logged
Tips and donations are much appreciated.
NXT: NXT-HPW9-EC4H-KJPT-7STQV (Hallmarked 100% solar powered cubietruck node (uptime 107 days!). If this is not Green, what else?)

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #152 on: May 23, 2014, 06:03:15 pm »

Nice to see some success again with MGW, thanks jl777!

Any eta when MGW will be ready for Mainnet? I think everyone is waiting for it.
I am struggling with the current method of tracking account balances. It works, but it is too complicated for me to make changes to it easily.
This is usually a sign that it need to be reworked.

Now I am very reluctant to make wholesale changes at this point, but the fact that after two days of trying to make a "simple" change (the three MGW servers dont agree on which withdrawal to do next and deadlocks) I am at the point of biting the bullet and reworking it.

So, I need some help and other eyes to make sure that after this reworking MGW balances will be totally accurate.

Now that I sped up reprocessing 100 times, I can consider more brute force methods as opposed to the current state machine method. As it is, any error anywhere, ever, would propagate forever. While my code has very low error rates, the forever propagation concerns me, especially if a HDD hiccup or network glitch can introduce an error.

Here is my plan.

Now that I have asset dividend calculations precise at any block, I can build on that. Internally, each NXT acct with an asset tx history has every asset tx that it ever did in a linear list. The number of transactions for any given user will be small compared to CPU speed, so linear is fine and so will recalculating everything from scratch.

Instead of a complex state machine that needs to make sure it has grabbed different events at different places in the right order to prevent any double transfers but to always make sure exactly one transfer is made, I can scan the list of asset tx and see if anything should be done. I think this greatly simplifies things as I can even now visualize the possible variations.

Ignoring how many confirmations a tx has, we have these trigger conditions:
A) external blockchain deposit but no corresponding NXT asset xfer
B) NXT asset withdrawal, but no external blockchain payment

Really it seems so simple now, I wonder why I had problems with this? So, I am concerned that I am missing some cases.

Since each NXT asset tx already has an entry and I even already defined a *cointxid ptr (unused) in the structure, I guess I knew something like this was needed. I already have the list of external blockchain deposits and it has fields to indicate completion, I think the simplifying factor here is that I will end up with just one list per NXT acct.

Since one of the three MGW will actually issue the asset xfer, the best way to sync the state of all three is for them to update the list with the binding of external blockchain deposit to NXT asset list when they see the asset xfer. Just need to make sure that the server that does the asset xfer doesnt do more than one. That means RAM based state variable for this and waiting N blocks on startup before doing any transfers to satisfy an external blockchain deposit.

If the above is good, we now have a totally synchronized state of deposits (after the asset xfer appears on the blockchain), during the time it is pending one of the MGW servers will have a different state. There is potential here of an "attack" by someone depositing all the time to get the three servers in a different state. To avoid this deposit attack, I think all I have to do is, nothing. As long as I am just scanning the list of the NXT acct's asset tx and dont credit any account until N blocks have passed, only what has appeared on the blockchain will affect the MGW's crediting of each account. The suppression of sending more than one transfer shouldnt affect the balance calculation, so I think I can ignore the "deposit attack"

OK, now I just need a deterministic way to select the NXT account to process. It seems a bit brute force, but 20,000 is a small number to a CPU and no asset has anywhere near that number of asset holders. I have a list of all asset tx for each asset, so I just scan that list for all NXT accts and scan each NXT acct's list for any required withdrawal processing. I need a timestamp of the block in which the withdrawal request came in (via asset xfer), then I can just process them in order of NXT acct #, or withdrawal size, doesnt matter as all three MGW servers should have the same lists

Once the three MGW servers are in agreement on the order of withdrawals that need to be processed, that solves the last sticking point. All external blockchain tx will map onto a NXT asset tx. I can verify that no tx is processed until N confirmations. I think processing one withdraw at a time will be fast enough as it can do several per minute, which is thousands of withdraws per day.

If the above is good, I can code it up and finally release it for testing.
Feedback please. MGW is super important and I want to have more than just me to verify the highlevel algo

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

valarmg

  • Hero Member
  • *****
  • Karma: +178/-57
  • Offline Offline
  • Posts: 1766
    • View Profile
Re: Multigateway status reports
« Reply #153 on: May 23, 2014, 08:06:39 pm »

Let me see if I can understand this. You only have two MGW trigger events. Deposit + Withdrawal. Deposit=Deposit from external coin and asset is added to NXT account. Withdrawal=Asset is withdrawn from AE, and coin is sent to external wallet.

A. Deposit.
Deposit request occurs (external request sent to MWG).
Multisig wallet is created, and funds are lodged in that wallet.
When confirmed, coin deposit details are entered into central list, agreed upon by the 3MGW.
One of the MGWs does the asset deposit, using a state machine to make sure it's only done once.

B Withdrawal.
The MGW is continually polling all the NXT addresses, looking for withdrawals from MGW assets.
When found, they are added to a central list, once again agreed by all 3 MGWs, order is also agreed.
Going through the list one by one, funds are sent from central MGW wallet back onto external blockchain.
Logged
NXT-CSED-4PK5-AR4V-6UB5V

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #154 on: May 23, 2014, 08:21:26 pm »

Let me see if I can understand this. You only have two MGW trigger events. Deposit + Withdrawal. Deposit=Deposit from external coin and asset is added to NXT account. Withdrawal=Asset is withdrawn from AE, and coin is sent to external wallet.

A. Deposit.
Deposit request occurs (external request sent to MWG).
Multisig wallet is created, and funds are lodged in that wallet.
When confirmed, coin deposit details are entered into central list, agreed upon by the 3MGW.
One of the MGWs does the asset deposit, using a state machine to make sure it's only done once.

B Withdrawal.
The MGW is continually polling all the NXT addresses, looking for withdrawals from MGW assets.
When found, they are added to a central list, once again agreed by all 3 MGWs, order is also agreed.
Going through the list one by one, funds are sent from central MGW wallet back onto external blockchain.
There is a third significant event, the request for deposit addresses. until someone gets a deposit address, MGW basically cant do anything for them. The user publishes AM that requests a multisig deposit addr, the MGW servers coordinate and one of them pubishes AM with the multisig deposit addr

Now, MGW is also monitoring all bitcoind blocks (and LTC, DOGE, DRK, etc) and when it sees a multisig address it has issued, this triggers a transfer of assets to the user's acct corresponding with the deposit

I just take the (NXT addr % NUM_GATEWAYS) to choose which gateway does the actual asset transfer, but all three are independently doing all the other calculations. They all have independently updated asset lists and NXT acct specific asset tx list

MGW does not need to monitor all NXT addr, just all NXT blocks and whenever there is a transfer of a BTC asset back to the MGW, this creates a withdrawal request. Hopefully the asset xfer that did this has a comment field validating destination, if not it will default to an address that was registered when requesting multisig deposit address

The reason why I dont think MGW will be a tempting target for hackers is that there is NO CENTRAL WALLET! Internally I create a virtual wallet that is composed of the sum of all unspent outputs from all the depositor's multisig accts. That also means no sweep transaction, as soon as a deposit gets enough confirmations, the MGW virtual balance is adjusted.

Here comes the part that automatically establishes MGW consensus. Not sure if you know about rawtransactions and how to make them. Trust me, it is not as easy as it sounds. Every detail of the tx needs to be specified in the precise syntax. Then at the end of it you get a bunch of bytes that is the transaction. It is these bytes that all MGW servers need to agree on, exactly. So that means every detail of the tx is the same. This means the order of withdrawal processing is critical.You need a deterministic algo to define all the tx parameters and it has to end up with the identical rawtx bytes on all three MGW server.

To make it even less of a target, over time, all the MGW multisig deposit addresses will converge to the average balance. Even if someone deposits 100 BTC or 0.1 BTC, over time they will both end up with around the average deposit. I am guessing around 1 BTC, maybe a bit more.

So a hacker would have to somehow compromise servers that are independently managed and then construct the rawtx bytes, get them signed to be able to steal 1 BTC. With active monitoring by the other servers and nodes running the MGW client, even if this unlikely event happens,we will find out right away and send in the dobermans

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

valarmg

  • Hero Member
  • *****
  • Karma: +178/-57
  • Offline Offline
  • Posts: 1766
    • View Profile
Re: Multigateway status reports
« Reply #155 on: May 24, 2014, 11:26:27 am »

I'm confused on one point. From what I'm reading, you seem to be saying that a withdrawal request is initiated from the AE. Is that right? However, it seems to me that from within the AE, it's only possible to transfer the asset around between NXT accounts, and that to withdraw, you'll have to initiate a redeem request via the MGW gui.

Edit: Ok, I think I understand. If (within AE) you transfer the asset to the MGW NXT account, then that triggers a withdrawal request. If so, two questions. 1) Can't you just monitor the MWG NXT account looking for transactions, rather than all the NXT blocks? and 2) What happens if a NXT account that has never deposited does the withdrawal (the asset was just transferred to them), they'll then have no deposit address?
« Last Edit: May 24, 2014, 11:41:32 am by valarmg »
Logged
NXT-CSED-4PK5-AR4V-6UB5V

chanc3r

  • Hero Member
  • *****
  • Karma: +124/-50
  • Offline Offline
  • Posts: 1019
  • NXTInspect
    • View Profile
Re: Multigateway status reports
« Reply #156 on: May 24, 2014, 11:41:47 am »

I'm confused on one point. From what I'm reading, you seem to be saying that a withdrawal request is initiated from the AE. Is that right? However, it seems to me that from within the AE, it's only possible to transfer the asset around between NXT accounts, and that to withdraw, you'll have to initiate a redeem request via the MGW gui.

You would initiate a withdraw by sending the asset back to the ASSET ISSUER NXT account owned by MGW - in order to balance the MGW would need to transfer the real coins from the multi-sig wallet to your registered withdrawal address.
Logged
NXT: 29996814460165 (NXT-JTA7-B2QR-8BFC-2V222)
@imrimr @NXTinspect

valarmg

  • Hero Member
  • *****
  • Karma: +178/-57
  • Offline Offline
  • Posts: 1766
    • View Profile
Re: Multigateway status reports
« Reply #157 on: May 24, 2014, 11:45:09 am »

I'm confused on one point. From what I'm reading, you seem to be saying that a withdrawal request is initiated from the AE. Is that right? However, it seems to me that from within the AE, it's only possible to transfer the asset around between NXT accounts, and that to withdraw, you'll have to initiate a redeem request via the MGW gui.

You would initiate a withdraw by sending the asset back to the ASSET ISSUER NXT account owned by MGW - in order to balance the MGW would need to transfer the real coins from the multi-sig wallet to your registered withdrawal address.

Thanks. (See my edit above, I have two further questions)
1) Can't you just monitor the MWG NXT account looking for transactions, rather than all the NXT blocks? and 2) What happens if a NXT account that has never deposited does the withdrawal (the asset was just transferred to them), they'll then have no deposit address?
Logged
NXT-CSED-4PK5-AR4V-6UB5V

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #158 on: May 24, 2014, 02:35:32 pm »

I'm confused on one point. From what I'm reading, you seem to be saying that a withdrawal request is initiated from the AE. Is that right? However, it seems to me that from within the AE, it's only possible to transfer the asset around between NXT accounts, and that to withdraw, you'll have to initiate a redeem request via the MGW gui.

You would initiate a withdraw by sending the asset back to the ASSET ISSUER NXT account owned by MGW - in order to balance the MGW would need to transfer the real coins from the multi-sig wallet to your registered withdrawal address.

Thanks. (See my edit above, I have two further questions)
1) Can't you just monitor the MWG NXT account looking for transactions, rather than all the NXT blocks? and 2) What happens if a NXT account that has never deposited does the withdrawal (the asset was just transferred to them), they'll then have no deposit address?
In scenarios where there are more than a certain number of accounts, it is much less work to just decode each block and all the tx inside it than poll every NXT acct to see if it got any changes.
Additionally NXTservices is tracking all assets, so much simpler and less API calls to just process all blocks

Withdrawals MUST be done from a preregistered NXT acct, the one that requested the multisig address
The withdraw function currently creates a tokenized request that can only be created by the NXT acct that is requesting the withdraw. Otherwise anybody can withdraw anybody elses funds. I thought that would be bad.

So that is why it is a two step process. You click the button to get the URL and paste it into the browser. Of course, we need a GUI to automate this, but private key always stays local.
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

lyynx

  • Jr. Member
  • **
  • Karma: +17/-0
  • Offline Offline
  • Posts: 83
    • View Profile
    • CoinWith.me
Re: Multigateway status reports
« Reply #159 on: May 26, 2014, 01:30:52 pm »

Keep up the great work, I think this will be one of the biggest additions to Nxt.
Logged
NxtDex.com - Directory of Nxt Related Assets, Shops and Services.
Pages: 1 ... 6 7 [8] 9 10 11  All
 

elective-stereophonic
elective-stereophonic
assembly
assembly