elective-stereophonic
elective-stereophonic
Multigateway status reports singapore
Please login or register.

Login with username, password and session length
Advanced search  

News:

Latest Stable Nxt Client: Nxt 1.12.1 Upgrade before block 2870000 is mandatory!

Pages: [1] 2 3 ... 11  All

Author Topic: Multigateway status reports  (Read 36905 times)

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Multigateway status reports
« on: March 24, 2014, 06:30:11 pm »

I will post multigateway status reports here

Current status is that I have the deposit side almost fully coded, but have to wait for "comment" field in assetTransfer to be able to fully debug it. I think I will put a temporary first fit comment generator so I can debug the actual deposit process. I also need to generalize synchronizing data to simplify coordination of the gateway servers.

James
« Last Edit: March 24, 2014, 06:39:43 pm by farl4bit, Reason: Capitalized first letter of subject »
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

bitcoinpaul

  • Hero Member
  • *****
  • Karma: +590/-590
  • Offline Offline
  • Posts: 3097
  • Karmageddon
    • View Profile
Re: multigateway status reports
« Reply #1 on: March 24, 2014, 06:32:01 pm »

Machine!
Logged
Like my Avatar? Reply now! NXT-M5JR-2L5Z-CFBP-8X7P3

VanBreuk

  • Hero Member
  • *****
  • Karma: +362/-19
  • Offline Offline
  • Posts: 2772
    • View Profile
Re: multigateway status reports
« Reply #2 on: March 24, 2014, 06:38:55 pm »

James

THIS. This helps a lot to feel just like home.

(and maybe the occasional white/grey cube in the avatar column... and that HOPPE red and blue one...)
« Last Edit: March 24, 2014, 07:03:00 pm by VanBreuk »
Logged
GPG Fingerprint: B020 D1C1 F289 3B2C 3577  9EAD 455D D175 5913 C7F1

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #3 on: March 25, 2014, 04:08:05 am »

some progress...
Have BTC, LTC, PPC, DOGE and DRK daemons running on three gateway servers
have corresponding wallets synced so I can test deposits
have platform code processing historical and realtime NXT blocks using three threads, 1 for each and a serializer for hash table updates
have five threads, one for each coin, also running. Finally got all the threads happy with each other and now I can debug code issues, instead of system issues

generating and processing new AM format, small binary header + compressed JSON
communicating with all 5 coin daemons properly, still have to debug each and every command's parsing, just drudgery no worries here
I broke the point to point network during refactoring, so have to get it working again and then debug the virtualized functions that execute on the remote server

Hmmmm, no wonder I am a bit tired! Anyway, going to rest for a bit. The networking bug is some silly thing I should be able to fix quickly once I get rested.

I have hashtables for all things that can have a lot of entries, like NXTaccts and txids. Each NXT acct has a list of assets, each with a linear list of all tx's. This will allow for reasonably quick access of any asset related query.

It feels like about two days away from getting the deposit side solidified. Still not completely sure how to properly deal with withdraws as far as updating the fragmented accounts, I hope I dont have to send multiple payments due to restrictions of fitting into standard transactions

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jabo38

  • Sr. Member
  • ****
  • Karma: +40/-38
  • Offline Offline
  • Posts: 381
    • View Profile
Re: Multigateway status reports
« Reply #4 on: March 25, 2014, 11:42:04 am »

some progress...
Have BTC, LTC, PPC, DOGE and DRK daemons running on three gateway servers
have corresponding wallets synced so I can test deposits
have platform code processing historical and realtime NXT blocks using three threads, 1 for each and a serializer for hash table updates
have five threads, one for each coin, also running. Finally got all the threads happy with each other and now I can debug code issues, instead of system issues

generating and processing new AM format, small binary header + compressed JSON
communicating with all 5 coin daemons properly, still have to debug each and every command's parsing, just drudgery no worries here
I broke the point to point network during refactoring, so have to get it working again and then debug the virtualized functions that execute on the remote server

Hmmmm, no wonder I am a bit tired! Anyway, going to rest for a bit. The networking bug is some silly thing I should be able to fix quickly once I get rested.

I have hashtables for all things that can have a lot of entries, like NXTaccts and txids. Each NXT acct has a list of assets, each with a linear list of all tx's. This will allow for reasonably quick access of any asset related query.

It feels like about two days away from getting the deposit side solidified. Still not completely sure how to properly deal with withdraws as far as updating the fragmented accounts, I hope I dont have to send multiple payments due to restrictions of fitting into standard transactions

James

That is amazing man!!!!  I think you must be a really hard worker.  It is really appreciated. :-)
Logged
Never Enough Money

abuelau

  • Sr. Member
  • ****
  • Karma: +74/-1
  • Offline Offline
  • Posts: 461
    • View Profile
    • mynxt.info
Re: Multigateway status reports
« Reply #5 on: March 25, 2014, 01:10:00 pm »

some progress...
Have BTC, LTC, PPC, DOGE and DRK daemons running on three gateway servers
have corresponding wallets synced so I can test deposits
have platform code processing historical and realtime NXT blocks using three threads, 1 for each and a serializer for hash table updates
have five threads, one for each coin, also running. Finally got all the threads happy with each other and now I can debug code issues, instead of system issues

generating and processing new AM format, small binary header + compressed JSON
communicating with all 5 coin daemons properly, still have to debug each and every command's parsing, just drudgery no worries here
I broke the point to point network during refactoring, so have to get it working again and then debug the virtualized functions that execute on the remote server

Hmmmm, no wonder I am a bit tired! Anyway, going to rest for a bit. The networking bug is some silly thing I should be able to fix quickly once I get rested.

I have hashtables for all things that can have a lot of entries, like NXTaccts and txids. Each NXT acct has a list of assets, each with a linear list of all tx's. This will allow for reasonably quick access of any asset related query.

It feels like about two days away from getting the deposit side solidified. Still not completely sure how to properly deal with withdraws as far as updating the fragmented accounts, I hope I dont have to send multiple payments due to restrictions of fitting into standard transactions

James

James,

I know we spoke about this briefly about this before but could you post the use cases for this feature here in this thread? That would be quite useful.

gaman

  • Full Member
  • ***
  • Karma: +1/-0
  • Offline Offline
  • Posts: 116
    • View Profile
Re: Multigateway status reports
« Reply #6 on: March 25, 2014, 06:34:49 pm »

where can we test your product ?
Logged

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #7 on: March 25, 2014, 06:49:24 pm »

where can we test your product ?
I am planning on an early beta release this week, i will post here. I am waiting for jean-luc to add a comment field to asset transfers so I can correlate external deposits with the corresponding NXT asset transfer. Without an explicit linkage, I will have to use an implicit one, like first fit. While it will probably be fine, I prefer knowing for sure what exact asset transfers were triggered by a specific deposit

I am still debugging the deposit side and the withdrawal side isnt fully refactored yet, but it is only Tuesday

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #8 on: March 25, 2014, 07:00:01 pm »

some progress...
Have BTC, LTC, PPC, DOGE and DRK daemons running on three gateway servers
have corresponding wallets synced so I can test deposits
have platform code processing historical and realtime NXT blocks using three threads, 1 for each and a serializer for hash table updates
have five threads, one for each coin, also running. Finally got all the threads happy with each other and now I can debug code issues, instead of system issues

generating and processing new AM format, small binary header + compressed JSON
communicating with all 5 coin daemons properly, still have to debug each and every command's parsing, just drudgery no worries here
I broke the point to point network during refactoring, so have to get it working again and then debug the virtualized functions that execute on the remote server

Hmmmm, no wonder I am a bit tired! Anyway, going to rest for a bit. The networking bug is some silly thing I should be able to fix quickly once I get rested.

I have hashtables for all things that can have a lot of entries, like NXTaccts and txids. Each NXT acct has a list of assets, each with a linear list of all tx's. This will allow for reasonably quick access of any asset related query.

It feels like about two days away from getting the deposit side solidified. Still not completely sure how to properly deal with withdraws as far as updating the fragmented accounts, I hope I dont have to send multiple payments due to restrictions of fitting into standard transactions

James

James,

I know we spoke about this briefly about this before but could you post the use cases for this feature here in this thread? That would be quite useful.
The primary use case for multigateway is to be able to automatically deposit funds and trade bitcoins (and any bitcoind fork) within AE.

All the deposit accounts are custom multisig accounts, no large sweep account like at the centralized exchanges. Funds are never under control of one entity, 2 of 3 multisig is used. The reason it is not 3 of 3 is to deal with the event one of the gateways goes missing, dont want to lose access to funds in that case. The other two would transfer to new acct, if something like that ever happened.

For beta release, I will have everything on three servers, but it is designed so that during the hardening phase, I can have it randomly pick 3 out of a much larger set of servers. Nobody will know exactly what three servers actually controls the acct, until after the account is created. Not sure if this level is needed.

This means no single large juicy target for hackers. All the accounts will be publicly monitored and any strangeness will be immediately detected and with the wallet fragmentation, any damage should be minimal.

Withdrawals are automated too, just transfer asset back to gateway and it will trigger a withdrawal process. All three gateways need to come to consensus on the exact withdrawal transaction and then and only then will it get put into an authentication queue. If the user enabled 2FA, there will then be a google authenticator step before the withdraw is submitted to the network.

All of the key steps will be visible on the NXT blockchain or the bitcoin blockchain. No more wondering what happened to your deposit, or more importantly whether the exchange is solvent!

The multigateway is just the first step in a much larger vision I have for NXT. NXT AE by itself is really just internally moving numbers around. You need to link these assets to something real for NXT AE to have real world value. Building on the multigateway base, I will be adding ... well lets keep some stuff a little mystery for now :)

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

kodtycoon

  • Hero Member
  • *****
  • Karma: +43/-18
  • Offline Offline
  • Posts: 916
    • View Profile
Re: Multigateway status reports
« Reply #9 on: March 26, 2014, 12:03:06 am »

@james.. you are incredible! bravo sir.. bravo
Logged

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #10 on: March 26, 2014, 12:05:56 am »

@james.. you are incredible! bravo sir.. bravo
thanks!
always helps to get positive energy when battling a bunch of bugs.
for a simple C programmer, the NXT protocol approach is kind of complicated, so I am not so fast as usual...
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

swartzfeger

  • Hero Member
  • *****
  • Karma: +50/-1
  • Offline Offline
  • Posts: 611
  • I bent my wookie
    • View Profile
    • https://www.instagram.com/swartzfeger/
Re: Multigateway status reports
« Reply #11 on: March 26, 2014, 01:03:00 am »

@james.. you are incredible! bravo sir.. bravo
thanks!
always helps to get positive energy when battling a bunch of bugs.
for a simple C programmer, the NXT protocol approach is kind of complicated, so I am not so fast as usual...

James, you are Nxt's man-beast chimera.

.: thumbs up :.
Logged

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #12 on: March 26, 2014, 02:16:46 am »

Finally got the point to point network fixed and got all three servers to autogenerate the same multisig addr for a specific customer acct.
Now I can finally debug the txout parsing and after that, asset transfers will be easy

going back in for round 2
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jefdiesel

  • Hero Member
  • *****
  • Karma: +88/-77
  • Offline Offline
  • Posts: 1275
    • View Profile
Re: Multigateway status reports
« Reply #13 on: March 26, 2014, 02:23:58 am »

serious machine.

When SkyNet goes operational, James will be the one to blame ;)
Logged
Member of D.O.R.C.S., creators of Lyth - An Emergent Trading Game | Asset ID: 2318361924203311027

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #14 on: March 26, 2014, 09:52:59 am »

I came up with a way to simplify the gateway logic. Got a bit carried away and coded all but the main withdraw function. The one that has to figure out what unspent outputs to use for a withdraw request. So now have a lot of new code that will provide a full day of debugging tomorrow,but at least the entire deposit/withdraw cycle is now in sight.

nobody seems to know if it is possible to use multiple multisig inputs and still keep it a standard transaction. I sure hope so. Worst case is that a withdraw will create more than one bitcoin transaction (and txfee). I guess a small price to pay for the multisig protection.

Anyway, the gateway's main processing loop is pretty simple:

Code: [Select]
                (*cp->add_unique_txids)(cp);                        // update universe of possible txids
                process_pingpong_queue(&cp->depositaddrs,cp);       // poll depositaddrs, queue to confirmed deposit

                cp->totalunspent = cp->totalspent = cp->maxunspent = cp->minunspent = 0;
                cp->minatp = cp->maxatp = 0;
                process_pingpong_queue(&cp->funds,cp);              // prune spent txids, calc totals
               
                process_pingpong_queue(&cp->withdrawreqs,cp);       // just wait for MIN_NXTCONFIRMS
                process_pingpong_queue(&cp->withdraws,cp);          // get consensus and submit to google_authenticator
                process_pingpong_queue(&cp->authenticate,cp);       // if 2FA enabled, do google_authenticator
                process_pingpong_queue(&cp->sendcoin,cp);           // submit rawtransaction and publish AM

I just process things from one queue to another. Each queue has a processing function associated with it and if it is is done with an entry, it goes to the next queue. Adding a new processing step becomes really easy. Not the most efficient thing, but considering the transaction times are measured in minutes, a few microseconds wasted is no biggie

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #15 on: March 27, 2014, 08:53:53 pm »

bitcoind bugs

At least dogecoind is not working like it says it should, or I didnt configure it right. Since multisig DOGE transactions are probably pretty low on the list of DOGE priorities, my guess is that I cannot rely on dogecoind to properly track multisig transactions. This means an entirely new module to parse and track raw bitcoind transactions...

Not sure how long it will delay things, but more than a day, thats for sure.

I keep hearing about P2SH transactions, but I am not sure if the bitcoind forks will support them at all. If anybody knows, I'd appreciate some pointers. The 2 of 3 multisig seems to be there for the vast majority of coins, so I would like to stick with that. However, based on some XCP vs BTC battles, some scenarios have multisig transactions being deprecated in the future! Not sure how credible the threat is, but clearly this affects the design of things.

I guess I could always virtualize the "multisigness" of each coin, but that would add more delays...

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #16 on: March 27, 2014, 09:41:30 pm »

pull on a thread...

Going to a parsing every block approach has made it so I need to revamp some internal data structures. I dont have to, but I am compelled to as it doesnt feel right to me when the data structures dont match what it should be. I guess my machine nature makes me reject my code that doesnt harmonize with the overall problem being solved

Thats the "bad" news. The good news is that now the reliance on external bitcoind daemons will be dramatically reduced. I originally relied on listreceivedbyaddress to actually work. What was I thinking?

Now, as long as the new blocks list all the transactions we will have a good dataset in my internal data structures so I can run simulations while I sleep

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #17 on: March 28, 2014, 05:26:27 am »

Finally got things reworked, at least on the deposit side. It is actually a lot cleaner and much less code. Just have one minor function to code before deposit side is feature complete, at least within the current iteration. Then the debugging begins, but hopefully with much less reliance on bitcoind, things will go much faster. In any case if there is a bug, it will be in my code which I can fix.

Just a little break before switching gears to debug mode. kind of spooky when all the compile and linker errors go away, especially after such a major rewrite. it feels like I am overlooking some loose ends, but thats what debugging is for.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

Fatih87SK

  • Hero Member
  • *****
  • Karma: +127/-36
  • Offline Offline
  • Posts: 2206
    • View Profile
Re: Multigateway status reports
« Reply #18 on: March 28, 2014, 05:38:22 am »

+21
Logged

nCtrl

  • Sr. Member
  • ****
  • Karma: +31/-0
  • Offline Offline
  • Posts: 254
    • View Profile
Re: Multigateway status reports
« Reply #19 on: March 28, 2014, 09:03:52 am »

I am sure these "daily reports" take away a chunck of your time, that maybe could better used but damn they are interesting, the difficulties you face, how you solve them etc. If you can, keep doing it, it is really interesting.
Logged
Pages: [1] 2 3 ... 11  All
 

elective-stereophonic
elective-stereophonic
assembly
assembly