elective-stereophonic
elective-stereophonic
Multigateway status reports  
Please login or register.

Login with username, password and session length
Advanced search  

News:

Latest Stable Nxt Client: Nxt 1.12.2

Pages: 1 2 3 ... 11 [All]

Author Topic: Multigateway status reports  (Read 42299 times)

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Multigateway status reports
« on: March 24, 2014, 06:30:11 pm »

I will post multigateway status reports here

Current status is that I have the deposit side almost fully coded, but have to wait for "comment" field in assetTransfer to be able to fully debug it. I think I will put a temporary first fit comment generator so I can debug the actual deposit process. I also need to generalize synchronizing data to simplify coordination of the gateway servers.

James
« Last Edit: March 24, 2014, 06:39:43 pm by farl4bit, Reason: Capitalized first letter of subject »
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

bitcoinpaul

  • Hero Member
  • *****
  • Karma: +590/-590
  • Offline Offline
  • Posts: 3097
  • Karmageddon
    • View Profile
Re: multigateway status reports
« Reply #1 on: March 24, 2014, 06:32:01 pm »

Machine!
Logged
Like my Avatar? Reply now! NXT-M5JR-2L5Z-CFBP-8X7P3

VanBreuk

  • Hero Member
  • *****
  • Karma: +362/-19
  • Offline Offline
  • Posts: 2772
    • View Profile
Re: multigateway status reports
« Reply #2 on: March 24, 2014, 06:38:55 pm »

James

THIS. This helps a lot to feel just like home.

(and maybe the occasional white/grey cube in the avatar column... and that HOPPE red and blue one...)
« Last Edit: March 24, 2014, 07:03:00 pm by VanBreuk »
Logged
GPG Fingerprint: B020 D1C1 F289 3B2C 3577  9EAD 455D D175 5913 C7F1

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #3 on: March 25, 2014, 04:08:05 am »

some progress...
Have BTC, LTC, PPC, DOGE and DRK daemons running on three gateway servers
have corresponding wallets synced so I can test deposits
have platform code processing historical and realtime NXT blocks using three threads, 1 for each and a serializer for hash table updates
have five threads, one for each coin, also running. Finally got all the threads happy with each other and now I can debug code issues, instead of system issues

generating and processing new AM format, small binary header + compressed JSON
communicating with all 5 coin daemons properly, still have to debug each and every command's parsing, just drudgery no worries here
I broke the point to point network during refactoring, so have to get it working again and then debug the virtualized functions that execute on the remote server

Hmmmm, no wonder I am a bit tired! Anyway, going to rest for a bit. The networking bug is some silly thing I should be able to fix quickly once I get rested.

I have hashtables for all things that can have a lot of entries, like NXTaccts and txids. Each NXT acct has a list of assets, each with a linear list of all tx's. This will allow for reasonably quick access of any asset related query.

It feels like about two days away from getting the deposit side solidified. Still not completely sure how to properly deal with withdraws as far as updating the fragmented accounts, I hope I dont have to send multiple payments due to restrictions of fitting into standard transactions

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jabo38

  • Sr. Member
  • ****
  • Karma: +40/-38
  • Offline Offline
  • Posts: 381
    • View Profile
Re: Multigateway status reports
« Reply #4 on: March 25, 2014, 11:42:04 am »

some progress...
Have BTC, LTC, PPC, DOGE and DRK daemons running on three gateway servers
have corresponding wallets synced so I can test deposits
have platform code processing historical and realtime NXT blocks using three threads, 1 for each and a serializer for hash table updates
have five threads, one for each coin, also running. Finally got all the threads happy with each other and now I can debug code issues, instead of system issues

generating and processing new AM format, small binary header + compressed JSON
communicating with all 5 coin daemons properly, still have to debug each and every command's parsing, just drudgery no worries here
I broke the point to point network during refactoring, so have to get it working again and then debug the virtualized functions that execute on the remote server

Hmmmm, no wonder I am a bit tired! Anyway, going to rest for a bit. The networking bug is some silly thing I should be able to fix quickly once I get rested.

I have hashtables for all things that can have a lot of entries, like NXTaccts and txids. Each NXT acct has a list of assets, each with a linear list of all tx's. This will allow for reasonably quick access of any asset related query.

It feels like about two days away from getting the deposit side solidified. Still not completely sure how to properly deal with withdraws as far as updating the fragmented accounts, I hope I dont have to send multiple payments due to restrictions of fitting into standard transactions

James

That is amazing man!!!!  I think you must be a really hard worker.  It is really appreciated. :-)
Logged
Never Enough Money

abuelau

  • Sr. Member
  • ****
  • Karma: +74/-1
  • Offline Offline
  • Posts: 461
    • View Profile
    • mynxt.info
Re: Multigateway status reports
« Reply #5 on: March 25, 2014, 01:10:00 pm »

some progress...
Have BTC, LTC, PPC, DOGE and DRK daemons running on three gateway servers
have corresponding wallets synced so I can test deposits
have platform code processing historical and realtime NXT blocks using three threads, 1 for each and a serializer for hash table updates
have five threads, one for each coin, also running. Finally got all the threads happy with each other and now I can debug code issues, instead of system issues

generating and processing new AM format, small binary header + compressed JSON
communicating with all 5 coin daemons properly, still have to debug each and every command's parsing, just drudgery no worries here
I broke the point to point network during refactoring, so have to get it working again and then debug the virtualized functions that execute on the remote server

Hmmmm, no wonder I am a bit tired! Anyway, going to rest for a bit. The networking bug is some silly thing I should be able to fix quickly once I get rested.

I have hashtables for all things that can have a lot of entries, like NXTaccts and txids. Each NXT acct has a list of assets, each with a linear list of all tx's. This will allow for reasonably quick access of any asset related query.

It feels like about two days away from getting the deposit side solidified. Still not completely sure how to properly deal with withdraws as far as updating the fragmented accounts, I hope I dont have to send multiple payments due to restrictions of fitting into standard transactions

James

James,

I know we spoke about this briefly about this before but could you post the use cases for this feature here in this thread? That would be quite useful.

gaman

  • Full Member
  • ***
  • Karma: +1/-0
  • Offline Offline
  • Posts: 116
    • View Profile
Re: Multigateway status reports
« Reply #6 on: March 25, 2014, 06:34:49 pm »

where can we test your product ?
Logged

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #7 on: March 25, 2014, 06:49:24 pm »

where can we test your product ?
I am planning on an early beta release this week, i will post here. I am waiting for jean-luc to add a comment field to asset transfers so I can correlate external deposits with the corresponding NXT asset transfer. Without an explicit linkage, I will have to use an implicit one, like first fit. While it will probably be fine, I prefer knowing for sure what exact asset transfers were triggered by a specific deposit

I am still debugging the deposit side and the withdrawal side isnt fully refactored yet, but it is only Tuesday

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #8 on: March 25, 2014, 07:00:01 pm »

some progress...
Have BTC, LTC, PPC, DOGE and DRK daemons running on three gateway servers
have corresponding wallets synced so I can test deposits
have platform code processing historical and realtime NXT blocks using three threads, 1 for each and a serializer for hash table updates
have five threads, one for each coin, also running. Finally got all the threads happy with each other and now I can debug code issues, instead of system issues

generating and processing new AM format, small binary header + compressed JSON
communicating with all 5 coin daemons properly, still have to debug each and every command's parsing, just drudgery no worries here
I broke the point to point network during refactoring, so have to get it working again and then debug the virtualized functions that execute on the remote server

Hmmmm, no wonder I am a bit tired! Anyway, going to rest for a bit. The networking bug is some silly thing I should be able to fix quickly once I get rested.

I have hashtables for all things that can have a lot of entries, like NXTaccts and txids. Each NXT acct has a list of assets, each with a linear list of all tx's. This will allow for reasonably quick access of any asset related query.

It feels like about two days away from getting the deposit side solidified. Still not completely sure how to properly deal with withdraws as far as updating the fragmented accounts, I hope I dont have to send multiple payments due to restrictions of fitting into standard transactions

James

James,

I know we spoke about this briefly about this before but could you post the use cases for this feature here in this thread? That would be quite useful.
The primary use case for multigateway is to be able to automatically deposit funds and trade bitcoins (and any bitcoind fork) within AE.

All the deposit accounts are custom multisig accounts, no large sweep account like at the centralized exchanges. Funds are never under control of one entity, 2 of 3 multisig is used. The reason it is not 3 of 3 is to deal with the event one of the gateways goes missing, dont want to lose access to funds in that case. The other two would transfer to new acct, if something like that ever happened.

For beta release, I will have everything on three servers, but it is designed so that during the hardening phase, I can have it randomly pick 3 out of a much larger set of servers. Nobody will know exactly what three servers actually controls the acct, until after the account is created. Not sure if this level is needed.

This means no single large juicy target for hackers. All the accounts will be publicly monitored and any strangeness will be immediately detected and with the wallet fragmentation, any damage should be minimal.

Withdrawals are automated too, just transfer asset back to gateway and it will trigger a withdrawal process. All three gateways need to come to consensus on the exact withdrawal transaction and then and only then will it get put into an authentication queue. If the user enabled 2FA, there will then be a google authenticator step before the withdraw is submitted to the network.

All of the key steps will be visible on the NXT blockchain or the bitcoin blockchain. No more wondering what happened to your deposit, or more importantly whether the exchange is solvent!

The multigateway is just the first step in a much larger vision I have for NXT. NXT AE by itself is really just internally moving numbers around. You need to link these assets to something real for NXT AE to have real world value. Building on the multigateway base, I will be adding ... well lets keep some stuff a little mystery for now :)

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

kodtycoon

  • Hero Member
  • *****
  • Karma: +43/-18
  • Offline Offline
  • Posts: 916
    • View Profile
Re: Multigateway status reports
« Reply #9 on: March 26, 2014, 12:03:06 am »

@james.. you are incredible! bravo sir.. bravo
Logged

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #10 on: March 26, 2014, 12:05:56 am »

@james.. you are incredible! bravo sir.. bravo
thanks!
always helps to get positive energy when battling a bunch of bugs.
for a simple C programmer, the NXT protocol approach is kind of complicated, so I am not so fast as usual...
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

swartzfeger

  • Hero Member
  • *****
  • Karma: +50/-1
  • Offline Offline
  • Posts: 611
  • I bent my wookie
    • View Profile
    • https://www.instagram.com/swartzfeger/
Re: Multigateway status reports
« Reply #11 on: March 26, 2014, 01:03:00 am »

@james.. you are incredible! bravo sir.. bravo
thanks!
always helps to get positive energy when battling a bunch of bugs.
for a simple C programmer, the NXT protocol approach is kind of complicated, so I am not so fast as usual...

James, you are Nxt's man-beast chimera.

.: thumbs up :.
Logged

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #12 on: March 26, 2014, 02:16:46 am »

Finally got the point to point network fixed and got all three servers to autogenerate the same multisig addr for a specific customer acct.
Now I can finally debug the txout parsing and after that, asset transfers will be easy

going back in for round 2
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jefdiesel

  • Hero Member
  • *****
  • Karma: +88/-77
  • Offline Offline
  • Posts: 1275
    • View Profile
Re: Multigateway status reports
« Reply #13 on: March 26, 2014, 02:23:58 am »

serious machine.

When SkyNet goes operational, James will be the one to blame ;)
Logged
Member of D.O.R.C.S., creators of Lyth - An Emergent Trading Game | Asset ID: 2318361924203311027

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #14 on: March 26, 2014, 09:52:59 am »

I came up with a way to simplify the gateway logic. Got a bit carried away and coded all but the main withdraw function. The one that has to figure out what unspent outputs to use for a withdraw request. So now have a lot of new code that will provide a full day of debugging tomorrow,but at least the entire deposit/withdraw cycle is now in sight.

nobody seems to know if it is possible to use multiple multisig inputs and still keep it a standard transaction. I sure hope so. Worst case is that a withdraw will create more than one bitcoin transaction (and txfee). I guess a small price to pay for the multisig protection.

Anyway, the gateway's main processing loop is pretty simple:

Code: [Select]
                (*cp->add_unique_txids)(cp);                        // update universe of possible txids
                process_pingpong_queue(&cp->depositaddrs,cp);       // poll depositaddrs, queue to confirmed deposit

                cp->totalunspent = cp->totalspent = cp->maxunspent = cp->minunspent = 0;
                cp->minatp = cp->maxatp = 0;
                process_pingpong_queue(&cp->funds,cp);              // prune spent txids, calc totals
               
                process_pingpong_queue(&cp->withdrawreqs,cp);       // just wait for MIN_NXTCONFIRMS
                process_pingpong_queue(&cp->withdraws,cp);          // get consensus and submit to google_authenticator
                process_pingpong_queue(&cp->authenticate,cp);       // if 2FA enabled, do google_authenticator
                process_pingpong_queue(&cp->sendcoin,cp);           // submit rawtransaction and publish AM

I just process things from one queue to another. Each queue has a processing function associated with it and if it is is done with an entry, it goes to the next queue. Adding a new processing step becomes really easy. Not the most efficient thing, but considering the transaction times are measured in minutes, a few microseconds wasted is no biggie

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #15 on: March 27, 2014, 08:53:53 pm »

bitcoind bugs

At least dogecoind is not working like it says it should, or I didnt configure it right. Since multisig DOGE transactions are probably pretty low on the list of DOGE priorities, my guess is that I cannot rely on dogecoind to properly track multisig transactions. This means an entirely new module to parse and track raw bitcoind transactions...

Not sure how long it will delay things, but more than a day, thats for sure.

I keep hearing about P2SH transactions, but I am not sure if the bitcoind forks will support them at all. If anybody knows, I'd appreciate some pointers. The 2 of 3 multisig seems to be there for the vast majority of coins, so I would like to stick with that. However, based on some XCP vs BTC battles, some scenarios have multisig transactions being deprecated in the future! Not sure how credible the threat is, but clearly this affects the design of things.

I guess I could always virtualize the "multisigness" of each coin, but that would add more delays...

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #16 on: March 27, 2014, 09:41:30 pm »

pull on a thread...

Going to a parsing every block approach has made it so I need to revamp some internal data structures. I dont have to, but I am compelled to as it doesnt feel right to me when the data structures dont match what it should be. I guess my machine nature makes me reject my code that doesnt harmonize with the overall problem being solved

Thats the "bad" news. The good news is that now the reliance on external bitcoind daemons will be dramatically reduced. I originally relied on listreceivedbyaddress to actually work. What was I thinking?

Now, as long as the new blocks list all the transactions we will have a good dataset in my internal data structures so I can run simulations while I sleep

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #17 on: March 28, 2014, 05:26:27 am »

Finally got things reworked, at least on the deposit side. It is actually a lot cleaner and much less code. Just have one minor function to code before deposit side is feature complete, at least within the current iteration. Then the debugging begins, but hopefully with much less reliance on bitcoind, things will go much faster. In any case if there is a bug, it will be in my code which I can fix.

Just a little break before switching gears to debug mode. kind of spooky when all the compile and linker errors go away, especially after such a major rewrite. it feels like I am overlooking some loose ends, but thats what debugging is for.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

Fatih87SK

  • Hero Member
  • *****
  • Karma: +127/-36
  • Offline Offline
  • Posts: 2206
    • View Profile
Re: Multigateway status reports
« Reply #18 on: March 28, 2014, 05:38:22 am »

+21
Logged

nCtrl

  • Sr. Member
  • ****
  • Karma: +31/-0
  • Offline Offline
  • Posts: 254
    • View Profile
Re: Multigateway status reports
« Reply #19 on: March 28, 2014, 09:03:52 am »

I am sure these "daily reports" take away a chunck of your time, that maybe could better used but damn they are interesting, the difficulties you face, how you solve them etc. If you can, keep doing it, it is really interesting.
Logged

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #20 on: March 28, 2014, 10:15:14 am »

I am sure these "daily reports" take away a chunck of your time, that maybe could better used but damn they are interesting, the difficulties you face, how you solve them etc. If you can, keep doing it, it is really interesting.
Only takes a few minutes, usually while I am waiting for some tests. Also, it actually helps me by writing stuff down.

Currently, I am getting all the little bugs out of the latest rev. A bit of a hiccup by changing to bitcoind_RPC as it is returning an extra layer of JSON, meaning the data coming from RPC is different than from just running bitcoind. Nothing structural, just have to get everything parsing properly. So far, no roadblocks seen, just tedious work

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #21 on: March 28, 2014, 11:29:17 pm »

Finally got bitcoind_RPC to properly submit createmultisig commands. So, about to get a stream of raw blocks to parse into my internal data structures.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #22 on: March 29, 2014, 04:01:38 am »

For people who complain about NXT not being finished, my feeling is that NXT API is much better than bitcoind as far as stability and speed goes.
At least with dogecoind, there are many RPC calls that take over a second! This is not good at all considering it is running on the same server...
I had thought there was a bug in my code looping a zillion times. Nope, just slow doge

Why the bitcoind JSON parameters are so finicky and inconsistent, it boggles the mind. Sometimes you can put "" around the argument, sometimes you must use them and other times you must not. Doesnt seem to be much rhyme or reason, just need to fiddle with each and every call to make sure it goes through.

So, now I am parsing every block at blazing speeds of 1 block per second. 99.9999% of the time waiting for the slow dogge. I think there must be some sort of curl setting I am not doing right though,it cant be this slow. It will take forever to sync up and I need it to sync up before I can test my code and find bugs.

So many issues not related to my code with this project. Everything goes in slow motion...

James

Edit: I take back the bitcoind is slow thing, http://stackoverflow.com/questions/17383089/libcurl-delays-for-1-second-before-uploading-data-command-line-curl-does-not
I got suspicious when the times were all right around 1000 milliseconds, seemed like it was clocked, like from a timeout. Sure enough the URL above had same problem and solution, now it is 1000 times faster! ~1 millisecond per bitcoind call, onto debugging my stuff!!
« Last Edit: March 29, 2014, 04:24:23 am by jl777 »
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #23 on: March 29, 2014, 05:07:21 am »

Now I am going too fast and bitcoind just randomly stops responding for a while. many seconds, "couldnt connect to server" error, even command line usage is broken during this time.

Again, I am not the first to push the performance limits: https://github.com/bitcoin/bitcoin/issues/2889

I guess 1000 commands per second was too good to be true, I will put in a 10 millisecond delay and hope that works

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #24 on: March 29, 2014, 05:53:36 am »

Now I am going too fast and bitcoind just randomly stops responding for a while. many seconds, "couldnt connect to server" error, even command line usage is broken during this time.

Again, I am not the first to push the performance limits: https://github.com/bitcoin/bitcoin/issues/2889

I guess 1000 commands per second was too good to be true, I will put in a 10 millisecond delay and hope that works

James
No reliable way to throttle the speed, other than to really slow it down.
I will just put in a 30 second delay on error, that seems to give the dogecoind enough time to recover from its broken state

It has reprocessed 5000+ blocks without any errors, I even saw the dreaded OP_RETURN which appeared as one of the vouts without any address or value. Triggered a warning message. The crazy thing is it was usually on a transaction with hundreds and hundreds of vouts, so the total size increase was less than 0.5% and this is in the super rare case that it happens. For all of this "bloat" the bitcoin devs are going to war against XCP.

bitcoin protocol now appears to be controlled by a dozen mining pools, some of which are controlled by bitcoin devs!
decentralized? Hmmm, not exactly.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #25 on: March 29, 2014, 06:03:35 am »

Confirmed deposit.0 DOGE.22431119b6aa6696f91393ff6c25ca3032d3093300a341588018dc918489a13a 1.23456788 -> NXT.11445347041779652448

!!!

Much excitement

The deposit I made that the dogecoind couldnt find for whatever reason, was found by brute force scanning all blocks. Now I can finally test the asset transfer code,which if it works means the deposit side is done. With comments being supported soon, I have a nice solution for specifying the withdraw address with each asset transfer. This way, there is no disconnect from whatever address is registered and where it ends up. It also means I wont need to database it either, this removes another point of failure

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

bitcoinpaul

  • Hero Member
  • *****
  • Karma: +590/-590
  • Offline Offline
  • Posts: 3097
  • Karmageddon
    • View Profile
Re: Multigateway status reports
« Reply #26 on: March 29, 2014, 07:13:53 am »

So many wow!
Logged
Like my Avatar? Reply now! NXT-M5JR-2L5Z-CFBP-8X7P3

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #27 on: March 29, 2014, 08:44:52 am »

So many wow!
Its slow going, but I now have all three servers pretty much in sync and even transferring assets. Just need to get the AM packet created properly so the two servers that didnt do the transfer can update their internal state.

A big issue is to be able to totally reconstruct 100% of multigateway state from scratch, by scanning all the different blockchains and have it exactly match on all three servers. Its harder than it sounds.

pretty much in the home stretch for DOGE, so tomorrow I can test the other coins individually and then all at the same time. So, beta testers, get ready for deposit testing early next week: BTC, LTC, DOGE and DRK

James

Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

CryptoScalper

  • Full Member
  • ***
  • Karma: +10/-24
  • Offline Offline
  • Posts: 192
  • Banned!
  • I DO NOT CONSENT
    • View Profile
Re: Multigateway status reports
« Reply #28 on: March 29, 2014, 09:36:51 am »

Confirmed deposit.0 DOGE.22431119b6aa6696f91393ff6c25ca3032d3093300a341588018dc918489a13a 1.23456788 -> NXT.11445347041779652448

!!!

Much excitement

James

BAM!    ;D


By the way...  as I also mentioned over at BTT in regard to nexern's latest API efforts...  I would like to recommend that you incorporate MetaTrader 4 (MT4)...  please not MT5  :-[...  as the official charting software of your Multigateway.  Don't let all that data from your servers go to waste!

MT4 comes with numerous indicators as it is...  I mean true Technical Analysis (TA) at the palm of your hands...  but the most interesting part is that it is fairly easy for a trader to code his/her own Custom Indicators using MT4's Algorithmic Language MQL4Trust me...  you can't beat it!    ;)

As you probably already know...  MT4 is widely distributed free by most Forex brokerages...  so using it will make your Multigateway (not to mention Nxt itself) very attractive to people with a better understanding of currency trading in general.
« Last Edit: March 29, 2014, 09:40:11 am by smaragda »
Logged
"A world with the money can not be perfect." - BCNext

Mistafreeze

  • Full Member
  • ***
  • Karma: +18/-3
  • Offline Offline
  • Posts: 199
  • I <3 Cheese
    • View Profile
Re: Multigateway status reports
« Reply #29 on: March 29, 2014, 01:54:18 pm »

This truly is so far beyond fantastic!!
Logged
Donate to my Beerfund NXT-L4WV-ZF8P-8X54-D6XML

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #30 on: March 30, 2014, 01:55:13 am »

ran into a nasty race condition bug where a deposit can get credited and transferred, but not seen in time in all cases
need to put some locks and queues and make sure the dataflows avoid most of the problems to minimize the race condition in the first case, but of course have to properly handle it.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #31 on: March 30, 2014, 01:56:43 am »

Confirmed deposit.0 DOGE.22431119b6aa6696f91393ff6c25ca3032d3093300a341588018dc918489a13a 1.23456788 -> NXT.11445347041779652448

!!!

Much excitement

James

BAM!    ;D


By the way...  as I also mentioned over at BTT in regard to nexern's latest API efforts...  I would like to recommend that you incorporate MetaTrader 4 (MT4)...  please not MT5  :-[...  as the official charting software of your Multigateway.  Don't let all that data from your servers go to waste!

MT4 comes with numerous indicators as it is...  I mean true Technical Analysis (TA) at the palm of your hands...  but the most interesting part is that it is fairly easy for a trader to code his/her own Custom Indicators using MT4's Algorithmic Language MQL4Trust me...  you can't beat it!    ;)

As you probably already know...  MT4 is widely distributed free by most Forex brokerages...  so using it will make your Multigateway (not to mention Nxt itself) very attractive to people with a better understanding of currency trading in general.
My understanding is that it is pretty expensive to get an MT4 license to run. However, I can think of ways of implementing a simplified MT4 on top of NXT,but that is for May
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

salsacz

  • Hero Member
  • *****
  • Karma: +239/-67
  • Offline Offline
  • Posts: 1762
  • NXT-R67P-6BZ2-XWAK-8RHZR
    • View Profile
Re: Multigateway status reports
« Reply #32 on: March 30, 2014, 03:23:58 am »

I don't know, where this belongs to: http://www.slideshare.net/mids106/ethereum-write-your-own-contracts - maybe here.
Logged
Holidays are (almost) over, check more news at: Nxt Academy

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #33 on: March 30, 2014, 05:46:14 am »

Finally got the race condition solved, I think
Now adding sanity checks on total and user balances, eg. lets not transfer more assets to user accts than the user has deposited, and make sure total assets transfers is never more than total deposited!

Multigateway will be partially hosted on mynxtcoin.org
I have feelers out to a couple more possible hosts, hopefully will get the required three soon

Now that all the ticky tack bugs seem to be fixed and things are actually doing what they are supposed to, its time to write new bugs :)
Onto withdraws as I am not in the mood for changing to other coins, DOGE all the way to feature completeness!

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

CryptoScalper

  • Full Member
  • ***
  • Karma: +10/-24
  • Offline Offline
  • Posts: 192
  • Banned!
  • I DO NOT CONSENT
    • View Profile
Re: Multigateway status reports
« Reply #34 on: March 30, 2014, 06:38:07 am »

My understanding is that it is pretty expensive to get an MT4 license to run. However, I can think of ways of implementing a simplified MT4 on top of NXT,but that is for May

I am afraid you are correct...  it seems I confused all this free distribution of MT4 by Forex brokerages with the actual licensing   :-[...  from a brief search I saw figures up to $100,000 being thrown around.

Can't wait to see what you implement in May...  in June...  in July...  in August...     ;)
Logged
"A world with the money can not be perfect." - BCNext

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #35 on: March 30, 2014, 09:34:01 am »

Well it feels like toward the end of doing a jigsaw puzzle. Still tricky, but the pieces keep falling into place. Its a good indication that my overall design is solving the problem at hand. It is always much easier the second time around, so the proof of concept version has been very useful, even though almost none of the code has survived the rewrite

Got the low level part of withdraw coded, so looking forward to the painstaking debugging process of getting every syntax perfect so bitcoind does gag. OK, I'm kidding about that. debugging is like hitting your head against the wall, over and over again. The only good thing about it is when it is done.

I am not sure the limits of what the various networks treat as standard transactions, so I need to get the send money part coded so I can characterize that. After I get it debugged tomorrow and find out what the restrictions are, I can map out a plan for managing all of the multisig accts. They start out as deposit accts, but as soon as the NXT asset transfer is done, they are put into a virtual unspent pool, which all the withdraws can use. I have a feeling I wont be able to do more than one multisig source account per withdraw, which isnt a problem if there is an account with a big enough balance. Problem is, I will be making sure there are no big balances, so there could well be multiple transactions needed per withdraw. All depends on each coins miners and what transactions they accept.

Until I know exactly what I need to do, I cant predict how long it will take. Having to write a block level parser delayed things by a couple days, but I am still kind of on schedule as deposits appear to work, at least in one case.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #36 on: March 31, 2014, 07:23:52 am »

Slow progress
It was taking so long to run each test as I had to reprocess a weeks worth of DOGE logs and dogecoind keeps stalling
so did a bunch of code cleanup and got a test framework that allows for much faster turnaround
currently testing the threeway synchronizing of the exact rawtransaction for a hardcoded withdraw. This is finding all sorts of small issues that just need to be fixed before I can even get a withdraw sent. So still not sure how many inputs I can use and stay as standard tx

Still left to do:
a) generating withdraw request from asset xfer, the hard part is to make sure to do it once and only once.
b) broadcasting a moneysent AM and using that to prevent sending money more than once
c) sanity testing with each of the supported coins
d) manually generate testing deposit addresses and <beta release>

e) google authenticator
f) some sort of automated account balancer to prevent any super large acct
g) instrumentation to allow for easy external tracking of balances
h) fix bugs

i) code review
j) release candidate

James
« Last Edit: March 31, 2014, 07:39:09 am by jl777 »
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #37 on: March 31, 2014, 09:34:34 am »

Took all day, but got a synchronized withdrawal signed and submitted. It helps to use the right command "sendrawtransaction" instead of "submitrawtransaction"!
Must be getting tired...
I got txid from dogecoind, but it is not appearing on the network.
probably need to pay the txfee
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #38 on: March 31, 2014, 10:12:43 am »

Just as I was waiting for testing the withdraw where i paid the txfee, I lost contact with all my servers :(
since its already tomorrow, i guess its a sign to call it a day and hope the servers are back when I wake up
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

bitcoinpaul

  • Hero Member
  • *****
  • Karma: +590/-590
  • Offline Offline
  • Posts: 3097
  • Karmageddon
    • View Profile
Re: Multigateway status reports
« Reply #39 on: March 31, 2014, 10:13:39 am »

Yeah, get some sleep.

See you in 3 hours.
Logged
Like my Avatar? Reply now! NXT-M5JR-2L5Z-CFBP-8X7P3

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #40 on: April 01, 2014, 12:29:38 am »

Got the tx confirmed by the network
now waiting on test results for multiple inputs being confirmed on network
However, I have found some nasty error cases that are possible due to not having comment field yet. Binding of NXT transfers with the other coin actions is thus not 100% guaranteed. Good thing we will get comment field soon!

Also, accounting shouldnt be so difficult, but getting all three servers to agree exactly on the exact balance of all accounts is actually not so easy, since one server usually does the actual transfer while the other two wait for confirmation via AM. Then there is the restoring state perfectly from a cold start.

Not the most fun type of stuff for me, but gotta make sure everything reconciles. Now that I finally have the multisig xfers actually working again, I feel the mood for getting this all sorted today. The best way is to simplify the code's control flow to limit the possibilities and then make sure all those cases are correct.

I expect a long night ahead...

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #41 on: April 01, 2014, 03:50:12 am »

I got a multi-input multisig tx to get confirmed, so that is really good news
now if I can get multiacct multiinput multisig to work, then account balance management wont be so bad
As it is I think I figured out how to get the accounts to converge to the average balance

The basic idea is to find the acct with the largest and smallest amount of unspent balance. Then satisfy the withdraw request from the largest unspent output(s) so that there is likely to be a lot of change. Then allocate the change so both the previous smallest and biggest would end up with the same new balance. Cant guarantee this will happen, as there is no assurance that there will be enough change to be able to balance the two accounts, but over time, as the acct balances converge to each other, the odds get better and better that the extreme accts can be matched up. Once that happens, before long all the acct balances should hover around the overall average, though I havent simulated this, it seems like a decent algo

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #42 on: April 01, 2014, 05:49:06 am »

Now I know why most all centralized exchanges use a sweep acct. That makes it easy to know whether an unspent txout is a user deposit or internal transfer.
I cant figure out how to determine with 100% accuracy at the moment this info is needed, eg. suppress spurious asset transfers!, without doing a horrible, horrible thing.

Based on all the uproar XCP's usage of OP_RETURN has generated, the only safe place I can think of to put the linkage is in an actual spendable output. Anything else is likely to be purged by the bitcoin devs who seem intent on controlling what goes into their blockchain. This means a small amount needs to be used for this. Assuming that people are ok with this, I can use this to get the multigateway to be self-sustaining. I am currently setting it to be .0001 of whatever coin, so it is really a small amount when it comes to anything but BTC.

If anybody has a problem with this fee, let me know

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

bitcoinpaul

  • Hero Member
  • *****
  • Karma: +590/-590
  • Offline Offline
  • Posts: 3097
  • Karmageddon
    • View Profile
Re: Multigateway status reports
« Reply #43 on: April 01, 2014, 06:49:53 am »

Set the fee and we talk in the future, if it needs to be changed.
Logged
Like my Avatar? Reply now! NXT-M5JR-2L5Z-CFBP-8X7P3

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #44 on: April 01, 2014, 09:33:06 am »

Saw the finish line and

left to do:
a) generating withdraw request from asset xfer, the hard part is to make sure to do it once and only once.
b) broadcasting a moneysent AM and using that to prevent sending money more than once
c) sanity testing with each of the supported coins
d) manually generate testing deposit addresses and <beta release>

e) google authenticator
f) some sort of automated account balancer to prevent any super large acct
g) instrumentation to allow for easy external tracking of balances
h) fix bugs

i) code review
j) release candidate

Basically I am "code complete" for hardcoded topology setups. Before production, I want to make it runtime configurable, but you can see from the list above, the remaining features (other than google authenticator) are debugging related. It ended up being a lot more new code than I expected, so probably a full day of debugging before beta release candidate.

I will make an initial beta release for advanced users and then write some automated tests to really pound on things before making a larger scale beta release.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #45 on: April 01, 2014, 10:43:27 pm »

comment field arrived!
Will work on integrating that in today. Well worth it as it makes correlations of NXT asset transfers 100% linkable to specific bitcoind txid. Plus a few other things :)

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

IveBeenBit

  • Full Member
  • ***
  • Karma: +7/-2
  • Offline Offline
  • Posts: 131
    • View Profile
Re: Multigateway status reports
« Reply #46 on: April 02, 2014, 10:26:57 pm »

subscribed
Logged
I need your suggestions for Nxt Podcast ideas. Click here and speak your mind!

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #47 on: April 02, 2014, 11:26:32 pm »

It seems that testnet is finally caught up regenerating, so I am able to resume debugging.
I did manage to get a really good idea during the down time.
Forming a team to make it real.
More news on that soon.

James

Edit: I've generated test addresses for everyone. I will test multiple address scenarios, but have to write instrumentation code so I can see what the multigateway is thinking. Much complexity.
« Last Edit: April 02, 2014, 11:31:12 pm by jl777 »
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #48 on: April 03, 2014, 01:38:57 am »

Just as I say that, 9.0e is out supporting nxtoshis!
gotta rework some code, but this is very good news

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #49 on: April 03, 2014, 08:32:48 am »

testnet still wonky...
did a total housecleaning on the code. Had to restructure it to remove as much knowledge of multigateway from the NXTprotocol code. This is so I can cleanly add more and more functionality without needing for each module to worry about all the others.

Anyway, it was a pretty major change. Just got it to compile and after one silly bug, it actually seems to be running. Hmmm. Expected more bugs.

As soon as testnet is stable, I can finally issue and test new acct numbers, but i still havent been able to get a test for multiple acct multiple input multisig.

Oh, the accounting reconciliation stuff. Always something left to do...

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #50 on: April 04, 2014, 12:06:58 am »

third day in a row testnet reset
no testNXT, so cant test the code
I feel the mood for coding the secret project today

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

Fatih87SK

  • Hero Member
  • *****
  • Karma: +127/-36
  • Offline Offline
  • Posts: 2206
    • View Profile
Re: Multigateway status reports
« Reply #51 on: April 04, 2014, 05:35:07 am »

What is your testnet account?

Found it.
« Last Edit: April 04, 2014, 05:59:05 am by Fatih87SK »
Logged

abctc

  • Hero Member
  • *****
  • Karma: +148/-13
  • Offline Offline
  • Posts: 1396
    • View Profile
Re: Multigateway status reports
« Reply #52 on: April 04, 2014, 06:21:53 am »

..   no testNXT, so cant test the code
- I've sent you 50k testNXT yesterday, 3.4.2014, at 09:04:02 GMT:

Quote
getTransaction:
transaction:  1537229809575493639
hash:   
{
    "referencedTransaction": "0",
    "confirmations": 627,
    "hash": "a67f9293395beb13077fbbe996951a82a6b85a8b78d14ad0caa51458cef669d2",
    "amountNQT": "5009900000000",
    "block": "15504955479361445353",
    "type": 0,
    "amountNXT": "50099",
    "feeNQT": "100000000",
    "recipient": "18232225178877143084",
    "sender": "14386024746077933238",
    "timestamp": 11220037,
    "subtype": 0,
    "feeNXT": "1",
    "senderPublicKey": "dc7b03d7dd03fe316bb3321c65683c7e528dbed6bdedd67d5aeae2e3dd170126",
    "deadline": 120,
    "blockTimestamp": 11221442,
    "signature": "e0700c66902306712d79464312543e181d6ea8f93fe12d5c6117fa618ca9350c60ce0ddf49e99007391a9ac458d41d1a527bb0ec40954d1507d2f0385c3136d8"
}

Please, let me know, if you need more testNXT.
But I have not so much of them, roughly 1M testNXT.
Logged
Welcome to the Nxt generation of crypto!   Magis quam Moneta (More than a Coin)
"Do not worry, it is an attack" (c) Jean-Luc

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #53 on: April 04, 2014, 07:19:40 am »

..   no testNXT, so cant test the code
- I've sent you 50k testNXT yesterday, 3.4.2014, at 09:04:02 GMT:

Quote
getTransaction:
transaction:  1537229809575493639
hash:   
{
    "referencedTransaction": "0",
    "confirmations": 627,
    "hash": "a67f9293395beb13077fbbe996951a82a6b85a8b78d14ad0caa51458cef669d2",
    "amountNQT": "5009900000000",
    "block": "15504955479361445353",
    "type": 0,
    "amountNXT": "50099",
    "feeNQT": "100000000",
    "recipient": "18232225178877143084",
    "sender": "14386024746077933238",
    "timestamp": 11220037,
    "subtype": 0,
    "feeNXT": "1",
    "senderPublicKey": "dc7b03d7dd03fe316bb3321c65683c7e528dbed6bdedd67d5aeae2e3dd170126",
    "deadline": 120,
    "blockTimestamp": 11221442,
    "signature": "e0700c66902306712d79464312543e181d6ea8f93fe12d5c6117fa618ca9350c60ce0ddf49e99007391a9ac458d41d1a527bb0ec40954d1507d2f0385c3136d8"
}

Please, let me know, if you need more testNXT.
But I have not so much of them, roughly 1M testNXT.
yes thanks! I now have testNXT, at least until next testnet rewind :)
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #54 on: April 05, 2014, 01:05:05 am »

Finally! No new testnet. I have testNXT. all the accounts funded and have assets.
Generated test addresses for everyone. just had to change "fee" to "feeNXT", I should change it to NQT

Then finally, finally, test multiacct multinput multisig withdraws
With a few days off from multigateway, I should actually have better eyes to see what silly things i did after allnighters
Yes, yes, have to put in the instrumentation. Not sure why I am finding it hard to get myself to do that...

James

P.S. secret project needs multigateway, so everything is connected
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

Fatih87SK

  • Hero Member
  • *****
  • Karma: +127/-36
  • Offline Offline
  • Posts: 2206
    • View Profile
Re: Multigateway status reports
« Reply #55 on: April 05, 2014, 07:20:21 am »

Exciting!
« Last Edit: April 05, 2014, 07:25:26 am by Fatih87SK »
Logged

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #56 on: April 05, 2014, 08:07:30 am »

so many little bugs. yikes, forgot that i was cut off right in the middle of a rewrite. almost have it finally doing automated verified withdraws, but only so much of this type of debugging I can do in a day, especially when super cool secret project is calling my name!

I will split my time between them

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

bitcoinpaul

  • Hero Member
  • *****
  • Karma: +590/-590
  • Offline Offline
  • Posts: 3097
  • Karmageddon
    • View Profile
Re: Multigateway status reports
« Reply #57 on: April 05, 2014, 08:28:45 am »

Don't get distracted. This 'product' is very important and huge!
Logged
Like my Avatar? Reply now! NXT-M5JR-2L5Z-CFBP-8X7P3

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #58 on: April 05, 2014, 08:40:08 am »

Don't get distracted. This 'product' is very important and huge!
It doesnt do me or anyone else any good, if I am not making good progress
there is a time for debugging
there is a time for coding
there is a time for designing
there is a time for brainstorming
there is no time for sleep

James

P.S. The secret project enhances multigateway, dont worry
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

kodtycoon

  • Hero Member
  • *****
  • Karma: +43/-18
  • Offline Offline
  • Posts: 916
    • View Profile
Re: Multigateway status reports
« Reply #59 on: April 05, 2014, 07:47:20 pm »

Don't get distracted. This 'product' is very important and huge!
It doesnt do me or anyone else any good, if I am not making good progress
there is a time for debugging
there is a time for coding
there is a time for designing
there is a time for brainstorming
there is no time for sleep


im am DYING
James

P.S. The secret project enhances multigateway, dont worry

I AM DYING TO KNOW WHAT IT IS!!!! :(
Logged

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #60 on: April 06, 2014, 12:13:49 am »

Don't get distracted. This 'product' is very important and huge!
It doesnt do me or anyone else any good, if I am not making good progress
there is a time for debugging
there is a time for coding
there is a time for designing
there is a time for brainstorming
there is no time for sleep

James

P.S. The secret project enhances multigateway, dont worry

I AM DYING TO KNOW WHAT IT IS!!!! :(
I hope to be able to announce significant details within a week

Today's tidbit is that Adam Levine, yes of Lets Talk bitcoin, has joined the team based on the idea and is also eagerly awaiting more details.

I have a couple more team members to assemble, alpha release to make to Wesley, then I will announce more details.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

AdamBLevine

  • Jr. Member
  • **
  • Karma: +5/-0
  • Offline Offline
  • Posts: 19
    • View Profile
Re: Multigateway status reports
« Reply #61 on: April 06, 2014, 12:21:41 am »

Happy to join the team (As an advisor until I understand how everything works and am convinced it's as brilliant as jl777 thinks it is.)
Logged

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #62 on: April 06, 2014, 02:10:58 am »

I've decided to put the multigateway interface into a websockets server. This will allow me to display account status and other info via API that is human readable and also easy for programs to access. I think this is why I had a hard time putting in all the instrumentation code for the multigateway. There was no easy way to actually look at the data.

This is why it is important not to force yourself, if something is holding you back, there usually is a reason. so now making requests to multigateway can be done via URL!

http://127.0.0.1:7777/multigateway?{"requestType":"register","DOGE":"<withdraw addr>","BTC":"<withdraw addr>"}

http://127.0.0.1:7777/multigateway?{"requestType":"withdraw","DOGE":"<any DOGE address>","amount":"12.456","TOTP":"123456"}

Now that I have a way to get commands in and out, I have a reference to code to and I also made it so that all the different NXTprotocol handlers I am writing plugs right into the websockets. Just change the "multigateway" to "<secret project>"

Now the one issue is that I am allergic to Windows. If anybody can help with getting the libwebsockets test-server compiled and running on Windows, that would be very helpful. Otherwise only Mac (and linux) users will be able to do the NXTprotocol API commands. I have made a few small changes to test-server.c, so as soon as someone can get a compile environment for libwebsockets http://libwebsockets.org/trac/libwebsockets, we can have a cross platform release

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #63 on: April 06, 2014, 03:07:38 am »

OK, I now have JSON commands from the browser. Surprising number of ways to mess up a simple pass through with so many layers.

Now I can actually make the JSON commands return useful info.

If anybody knows of a portable way to dynamically load and unload executable modules, let me know. It would be really nice to be able to swap in and out NXTprotocol handlers without having to recompile new configs

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #64 on: April 06, 2014, 03:35:54 am »

Do we need google authenticator? I think if I made it so that every request needs to have a NXT token generated, then it would prove the account holder (or at least someone with the password) is making the request. If someone else has the password, there is nothing multigateway can do anyway.

So, I propose making all important requests to be as follows:

http://127.0.0.1:7777/multigateway?[{"requestType":"important",...,"NXT":"1234567890123456789"},{"token":"<160 chars>"}], that way there is no ambiguity on what to tokenize. white space needs to be stripped. Just submit to NXT website token generator:

{"requestType":"important",...,"lastparm":"xyz"} as the website

I think this should be as secure as practical, essentially requiring account passphrase for all NXTprotocol API calls that are not just informational.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #65 on: April 06, 2014, 06:05:12 am »

Made a stripped down multigateway that can run locally. Seemed the most reliable way of verifying that the main multigateway servers are doing fine, plus I actually dont have a desktop version of Ubuntu, so only access to libwebsockets is on Mac.

Crammed two NXThandlers on top of NXTprotocol and both finally are reporting properly.

Now I can finally properly code the multigateway instrumentation and I can monitor it in realtime locally.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #66 on: April 06, 2014, 08:05:52 am »

How many people are running on Mac and available for testing.
Just putting in JSON URLs
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #67 on: April 06, 2014, 10:57:25 am »

400 lines of code later, multigateway JSON dump code is ready for debug. I just dump the entire internal state. once that is debugged, then it is a matter of making sure it stays self-consistent.

If you want to help, it would help if there was someone more than just me making sure all the JSON entries make sense. Its not so scary:

Code: [Select]
{
"gatewayid": -1,
"NXTaddr": "18232225178877143084",
"ipaddr": "181.47.159.125",
"NXTaccts": [{
"NXTaddr": "8737506522730586553",
"numassets": 1,
"assets": [{
"assetid": "15066647755760867565",
"quantity": "999.00000000",
"txids": [{
"txid": "14959687918642441123",
"sender": "16159101027034403504",
"receiver": "8737506522730586553",
"quantity": "999.00000000"
}]
}],
"numcoinaccts": 0
}, {
"NXTaddr": "2865886802744497404",
"numassets": 1,
"assets": [{
"assetid": "15066647755760867565",
"quantity": "998.00000000",
"txids": [{
"txid": "14682240803350035277",
"sender": "16159101027034403504",
"receiver": "2865886802744497404",
"quantity": "998.00000000"
}]
}],
"numcoinaccts": 0
}, {
"NXTaddr": "7126304194855053556",
"numassets": 0,
"numcoinaccts": 0
}, {
"NXTaddr": "16159101027034403504",
"numassets": 1,
"assets": [{
"assetid": "15066647755760867565",
"quantity": "-1997.00000000",
"txids": [{
"txid": "14959687918642441123",
"sender": "16159101027034403504",
"receiver": "8737506522730586553",
"quantity": "999.00000000"
}, {
"txid": "14682240803350035277",
"sender": "16159101027034403504",
"receiver": "2865886802744497404",
"quantity": "998.00000000"
}]
}],
"numcoinaccts": 0
}, {
"NXTaddr": "18232225178877143084",
"numassets": 0,
"numcoinaccts": 0
}, {
"NXTaddr": "1739068987193023818",
"numassets": 0,
"numcoinaccts": 0
}],
"coins": {
"DOGE": {
"assetname": "DOGE",
"funds": {
"deposits": "0.00000000",
"transfers": "0.00000000",
"withdraws": "0.00000000",
"moneysends": "0.00000000",
"remaining_transfers": "0.00000000",
"remaining_moneysends": "0.00000000",
"balance": "0.00000000"
},
"unspent": {
"maxunspent": "0.00000000",
"unspent": "0.00000000",
"maxavail": "0.00000000",
"minavail": "0.00000000",
"smallest_msigunspent": "0.00000000",
"confirmed": 0
},
"assetid": "13606501668992980954",
"serverport": "node2.mynxtcoin.org:22555",
"userpass": "dogecoinrpc:HSDU....",
"gateways": ["209.126.71.170", "209.126.73.156", "209.126.73.158"],
"coinid": 3,
"min_confirms": 3,
"pollseconds": 60,
"nummsigs": 0,
"numwithdraws": 0,
"numdeposits": 0,
"initdone": 0,
"blockheight": 170542,
"NXTheight": 0,
"txfee": "1.00010000",
"markeramount": "0.00010000"
},
"genaddr_txids": {
"numitems": 0,
"structsize": 224,
"hashsize": 100,
"numsearches": 0,
"numiterations": 0
},
"coin_txids": {
"numitems": 1,
"structsize": 112,
"hashsize": 100,
"numsearches": 2,
"numiterations": 2,
"numiterations": 1
},
"redeemtxids": {
"numitems": 0,
"structsize": 224,
"hashsize": 100,
"numsearches": 0,
"numiterations": 0
},
"gen depositaddr requests": {
"queuesize": 0
},
"call_gen_depositaddr": {
"queuesize": 0
},
"retryassetxfer": {
"queuesize": 0
},
"wait_for_NXTconfirms": {
"queuesize": 0
},
"process_withdraws": {
"queuesize": 0
},
"google authenticator": {
"queuesize": 0
},
"send coin": {
"queuesize": 0
}
}
}

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #68 on: April 06, 2014, 12:36:41 pm »

Just as I was waiting for test results we got assetoshis!
gotta do the whole beg for textNXT and reinitialize all the assets

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #69 on: April 06, 2014, 03:52:58 pm »

JSON outputs look decent, but cant test all the cases until testnet gets going again.
I did manage to get a generate deposits address (and withdraw address) with "website" key validation via websockets API, a two step process to generate the magic URL.

start with:
{%22requestType%22:%22genDepositaddrs%22,%22NXT%22:%2211445347041779652448%22,%22coins%22:{%22DOGE%22:%22DNyEii2mnvVL1BM1kTZkmKDcf8SFyqAX4Z%22}}

use the above to generate:
{%22token%22:%22bd855r204lqfskfg4que97uef8sh2blnh7ol9mcq2mmee1i2lu3a2ftnpqudlto04kqa15gpc14hu6c1dvs1p0vfjbkha5q423lqo8fbf5k0ketomirtkvveefqib5r03bl2pk2i3ju48dl7k4bpcqfehppch81h%22}

submit URL by combining:
http://127.0.0.1:7777/multigateway?[{%22requestType%22:%22genDepositaddrs%22,%22NXT%22:%2211445347041779652448%22,%22coins%22:{%22DOGE%22:%22DNyEii2mnvVL1BM1kTZkmKDcf8SFyqAX4Z%22}},{%22token%22:%22bd855r204lqfskfg4que97uef8sh2blnh7ol9mcq2mmee1i2lu3a2ftnpqudlto04kqa15gpc14hu6c1dvs1p0vfjbkha5q423lqo8fbf5k0ketomirtkvveefqib5r03bl2pk2i3ju48dl7k4bpcqfehppch81h%22}]

It looks much easier to read without %22 and with ", but somehow the browser is automatically converting.

anyway, this means that once I can get the libwebsockets compiled for Windows, there is no need for compiling a program as I will put in all the required controls in the API. Actually, other than being able to generate deposit addresses and set withdraw addresses is pretty much all the GUI the multigateway needs.

Next up is converting to assetoshis, and maybe I can finally get the withdraw side tested.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

kodtycoon

  • Hero Member
  • *****
  • Karma: +43/-18
  • Offline Offline
  • Posts: 916
    • View Profile
Re: Multigateway status reports
« Reply #70 on: April 06, 2014, 06:10:40 pm »

Just as I was waiting for test results we got assetoshis!
gotta do the whole beg for textNXT and reinitialize all the assets

James

james why not just ask who ever is restarting the testnet to send testnxt to an account and then pm you the passphrase.. just let it be the first thing they do after a restart so you dont loose time waiting for testnxt? or if you always have the same acc just get them to send it to you straight away as a priority after a restart? seems a silly thing to be loosing time over....
Logged

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #71 on: April 07, 2014, 01:49:47 am »

Just as I was waiting for test results we got assetoshis!
gotta do the whole beg for textNXT and reinitialize all the assets

James

james why not just ask who ever is restarting the testnet to send testnxt to an account and then pm you the passphrase.. just let it be the first thing they do after a restart so you dont loose time waiting for testnxt? or if you always have the same acc just get them to send it to you straight away as a priority after a restart? seems a silly thing to be loosing time over....
Forrest,
no worries, i do usually get testNXT pretty quickly
the bigger issue is that it seems to take about a day before testnet isnt wonky after a reset

However, this is why I like to have more than one project. I can always switch to a different project. Over the years I have found that it is impossible for everything a project needs to always be there exactly when you need it. It is like the difference between DOS (the microsoft OS remember that?) that could only do one task at a time. Sure it was 1% faster than a multitasking OS because it didnt have any overhead for switching processes, but start a disk I/O, wait forever,etc.

Unix type of multithreading always keeps the CPU busy.

Similarily, whenever the project I am working on has to wait for something (even myself if I cant figure out how to solve something), I switch. Some of the multigateway tests takes 10 minutes+ to run. If I just stared at the screen, i wouldnt be very productive, so I switch.

net result is that I am at least twice as productive as otherwise, but actually even more as the different projects enhance each other, so maybe it is more like a threefold boost. Then there is the hours I work, which is doubletime and I dont spend any time with meetings, commuting, etc, so that is another 2 to 3 times boost.

So, I am a simple C programmer with a 3*3 boost :)

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #72 on: April 07, 2014, 04:45:29 am »

Conversion to assetoshis is really simplifying things internally.
the need to be able to run a matching copy on user nodes that wont have write access to coin daemons is requiring me to convert a lot of internal functions to use coinid (just a number) instead of *cp (a pointer to coin daemon data structure). Since all the key transactions will appear in AM's, any NXT node will be able to keep track of any multigateway state changes.

Removing the assumption of having the coin daemon data is requiring going over all the code, but I had to do that due to assetoshis anyway.

Another long day in progress.

James

P.S. This Wed/Thu I will be offline due to real world stuff.
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

chanc3r

  • Hero Member
  • *****
  • Karma: +124/-50
  • Offline Offline
  • Posts: 1019
  • NXTInspect
    • View Profile
Re: Multigateway status reports
« Reply #73 on: April 07, 2014, 08:26:46 am »

I've tried to track James posts on this from BCT and here and summarise what the multi-sig gateway looks like so that I can understand it so comments on whether you think i've got it right would be most welcome because I don't want to waste James time to explain his genius to mere mortals like me.

This is my first attempt at it, I will continue this to further my own understanding but if useful to others I will continue to share.

https://dl.dropboxusercontent.com/u/53047281/multi-sig.doc

BR
Ian

PS I've also got a doc in progress on NodeCoin and NXT mixer but those are harder to decipher from James blogs ;)
Logged
NXT: 29996814460165 (NXT-JTA7-B2QR-8BFC-2V222)
@imrimr @NXTinspect

kodtycoon

  • Hero Member
  • *****
  • Karma: +43/-18
  • Offline Offline
  • Posts: 916
    • View Profile
Re: Multigateway status reports
« Reply #74 on: April 07, 2014, 09:30:35 am »

i dont know why but i keep saying ass-toshis in my head instead of asset-oshis! :L
Logged

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #75 on: April 07, 2014, 09:38:46 am »

I've tried to track James posts on this from BCT and here and summarise what the multi-sig gateway looks like so that I can understand it so comments on whether you think i've got it right would be most welcome because I don't want to waste James time to explain his genius to mere mortals like me.

This is my first attempt at it, I will continue this to further my own understanding but if useful to others I will continue to share.

https://dl.dropboxusercontent.com/u/53047281/multi-sig.doc

BR
Ian

PS I've also got a doc in progress on NodeCoin and NXT mixer but those are harder to decipher from James blogs ;)
You are very close!
With the change to assetoshis, there is no need for milliDOGEs anymore
also, for the pure multigateway functionality, all communication is via AM so there is no need for guardians. I really like your chart though! Maybe I will change how it works to match the chart :)

The guardians are more for the NXTmixer and NXTcash, though all these things tend to intermingle and who knows how it will evolve.

James

P.S. Make sure to checkout the API post, it should help you understand with the added details
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #76 on: April 07, 2014, 09:45:58 am »

The line count is creeping up...
I now have 12 files, just for the multigateway handling. This doesnt count the other 10 for NXTprotocol, nodecoin, etc.

Now that I reached code complete, it was time to totally rewrite it again :)

The need for remote nodes to run the multigateway wasnt exactly planned for and that broke an internal assumption, eg. that each computer running multigateway had access to the coin daemons. Now, there are three cases:

A) Full multigateway
B) Validating nodes, does almost all the Full multigateway does, expect for creating new accounts and sending money
C) Normal nodes, which just track things via AM and other NXT blockchain events

In order not to go insane, I need to make sure the same code is run in A) as in C) for the same function. The worst thing is to have two (or three) versions that do pretty much the same thing, but its in different places. One "harmless" change in one place and not the other and you end up debugging for hours.

So, this trigged a major rewrite. The good thing is that the low level code is porting intact, just the messy higher level code that ties it all together. Once all the redundancies have been squeezed out of the code, it becomes much harder for any bugs to remain hidden

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

chanc3r

  • Hero Member
  • *****
  • Karma: +124/-50
  • Offline Offline
  • Posts: 1019
  • NXTInspect
    • View Profile
Re: Multigateway status reports
« Reply #77 on: April 07, 2014, 10:57:59 am »


You are very close!
With the change to assetoshis, there is no need for milliDOGEs anymore
also, for the pure multigateway functionality, all communication is via AM so there is no need for guardians. I really like your chart though! Maybe I will change how it works to match the chart :)

The guardians are more for the NXTmixer and NXTcash, though all these things tend to intermingle and who knows how it will evolve.

James

P.S. Make sure to checkout the API post, it should help you understand with the added details

Hi James,

thanks for the reply and you like my picture :)
I thought I'd removed the fractions-workaround stuff now we have fractions.... I'll amend.
I've found the API thread on this board and that can will help to add / correct some detail.

I'd left the guardian server in because of this in the Infrastructure board request..

Quote from: jl777
I might be able to make a "thin guardian server" that could be run on people's NXT nodes. This would eliminate or reduce the need for a large number of guardian servers and still get a similar (if not better) effect.

So guardians are definitely gone for Multi-Sig-Gateway?

So I can get this straight (and partly with my inf-com hat on) how does a client connect to a gateway server ?
thus how do we ensure they all don't select the same one so that all the wallets don't end up on the same server pair?

Finally I like to add some state-transitions to show the flows between client / gateway-server and xCoind-server, as I think this would be useful, is there anywhere I can do this other than reading your code?
If that's the only way, as I can easily read 'C' is there anyway I can get the relevant code fragments?

Thanks,
Ian
Logged
NXT: 29996814460165 (NXT-JTA7-B2QR-8BFC-2V222)
@imrimr @NXTinspect

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #78 on: April 07, 2014, 12:25:10 pm »


You are very close!
With the change to assetoshis, there is no need for milliDOGEs anymore
also, for the pure multigateway functionality, all communication is via AM so there is no need for guardians. I really like your chart though! Maybe I will change how it works to match the chart :)

The guardians are more for the NXTmixer and NXTcash, though all these things tend to intermingle and who knows how it will evolve.

James

P.S. Make sure to checkout the API post, it should help you understand with the added details

Hi James,

thanks for the reply and you like my picture :)
I thought I'd removed the fractions-workaround stuff now we have fractions.... I'll amend.
I've found the API thread on this board and that can will help to add / correct some detail.

I'd left the guardian server in because of this in the Infrastructure board request..

Quote from: jl777
I might be able to make a "thin guardian server" that could be run on people's NXT nodes. This would eliminate or reduce the need for a large number of guardian servers and still get a similar (if not better) effect.

So guardians are definitely gone for Multi-Sig-Gateway?

So I can get this straight (and partly with my inf-com hat on) how does a client connect to a gateway server ?
thus how do we ensure they all don't select the same one so that all the wallets don't end up on the same server pair?

Finally I like to add some state-transitions to show the flows between client / gateway-server and xCoind-server, as I think this would be useful, is there anywhere I can do this other than reading your code?
If that's the only way, as I can easily read 'C' is there anyway I can get the relevant code fragments?

Thanks,
Ian
guardians aren't totally gone, but I cant say more without leaking info about the secret project...

Most of the gateway <-> client communication is done via AM, which can be generated via the websockets page.

If you can read C, then maybe you can do a source review? Its grown to ~5000 lines and its hard to see the obvious bugs with code you've been looking at for a long time.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

nCtrl

  • Sr. Member
  • ****
  • Karma: +31/-0
  • Offline Offline
  • Posts: 254
    • View Profile
Re: Multigateway status reports
« Reply #79 on: April 07, 2014, 12:29:19 pm »

I can do that as well as long as you let me know where I can find it. (I am on the phone, sorry if it is in plain sight)
« Last Edit: April 07, 2014, 12:30:50 pm by nCtrl »
Logged

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #80 on: April 07, 2014, 12:33:35 pm »

I can do that as well as long as you let me know where I can find it. (I am on the phone, sorry if it is in plain sight)
It is not public yet, want a closed review first, at least until it is deployed on mainnet.
I will be able to send you source directly

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #81 on: April 07, 2014, 12:35:24 pm »

Almost got all the compiler errors fixed. All the high level code got touched by the wave of changes. It is much cleaner and its good this happened. I can post the data structures

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #82 on: April 07, 2014, 12:38:15 pm »

Current data structures:

Code: [Select]
struct hashtable
{
    char *name;
    void **hashtable;
    uint64_t hashsize,numsearches,numiterations,numitems;
    unsigned long keyoffset,keysize,modifiedoffset,structsize;
};

struct hashpacket
{
    int32_t *createdflagp,doneflag,funcid;
    char *key;
    struct hashtable **hp_ptr;
    union { void *result; uint64_t hashval; };
};


struct NXT_guid
{
    struct NXT_str H;
    char guid[128 - sizeof(struct NXT_str)];
};

struct NXT_assettxid
{
    struct NXT_str H;
    uint64_t txidbits,assetbits,senderbits,receiverbits,quantity,price; // price 0 -> not buy/sell
    struct NXT_guid *duplicatehash;
    void *cointx;
    char *guid,*comment;
    int32_t hackattempts;
};

struct NXT_assettxid_list
{
    struct NXT_assettxid **txids;
    int32_t num,max;
};

struct NXT_asset
{
    struct NXT_str H;
    uint64_t assetbits;
    struct NXT_assettxid **txids;   // all transactions for this asset
    int32_t max,num;
};

struct NXT_acct
{
    struct NXT_str H;
    struct coin_acct *coinaccts;
    struct NXT_asset **assets;
    uint64_t *quantities;
    int32_t maxassets,numassets,numcoinaccts;
    struct NXT_assettxid_list **txlists;    // one list for each asset in acct
};


struct req_deposit
{
    uint64_t nxt64bits,modified,AMtxidbits;
    char NXTaddr[MAX_NXTADDR_LEN],NXTtxid[MAX_NXTADDR_LEN],cointxid[MAX_COINTXID_LEN];;
};

struct pubkey_info { char pubkey[128],coinaddr[64],server[64]; };

struct multisig_addr
{
    char NXTaddr[MAX_NXTADDR_LEN],addr[MAX_COINADDR_LEN],redeemScript[256];
    struct pubkey_info pubkeys[NUM_GATEWAYS];
    int32_t m,n,coinid,rippletag;
    int64_t maxunspent,unspent;
};

struct crosschain_NXTinfo
{
    uint64_t AMtxidbits,asset_txid;
    int32_t confirmed,pending;
};

struct crosschain_info
{
    uint64_t nxt64bits;
    int64_t value,assetoshis;
    struct multisig_addr *msig;
    struct coin_value *parent;
    int32_t NXTRTflag,NXTxfermask,coin_confirmed,isinternal;
    struct crosschain_NXTinfo NXT;
};

struct coin_value
{
    int64_t modified,value;
    struct crosschain_info *xp;
    struct coin_txid *parent,*spent,*pendingspend;
    union { char *script; char *coinbase; };
    int32_t parent_vout,spent_vin,pending_spendvin,isconfirmed,iscoinbase,isinternal;
    char coinaddr[MAX_COINADDR_LEN];
};

struct coin_txid
{
    int64_t modified,height;
    char txid[MAX_COINTXID_LEN];
    int32_t numvins,numvouts,hasinternal;
    struct coin_value **vins,**vouts;
};

struct rawtransaction
{
    char *destaddrs[4];
    struct coin_value *inputs[16];
    int64_t amount,change,inputsum,destamounts[4];
    int32_t numoutputs,numinputs;
    char rawtransaction[4096],signedtransaction[4096],txid[64];
};

struct withdraw_info
{
    struct server_request_header H;
    uint64_t AMtxidbits;
    int64_t amount,moneysent;
    int32_t coinid,srcgateway,destgateway,twofactor,authenticated;
    char cointxid[MAX_COINTXID_LEN],withdrawaddr[64],NXTaddr[MAX_NXTADDR_LEN],redeemtxid[MAX_NXTADDR_LEN],comment[1024];
    struct rawtransaction rawtx;
};

struct unspent_info
{
    int64_t maxunspent,unspent,maxavail,minavail,smallest_msigunspent;
    struct coin_value **vps,*maxvp,*minvp;
    int32_t maxvps,num;
    char smallest_msig[128];
};

struct balance_info { int64_t deposits,transfers,withdraws,moneysends,remaining_transfers,remaining_moneysends,balance; };

struct coin_acct
{
    struct balance_info funds;
    struct multisig_addr **msigs;
    int32_t enable_2FA,nummsigs,coinid;
    char withdrawaddr[MAX_COINADDR_LEN];
};

struct replicated_coininfo // runs on all nodes and only needs NXT blockchain
{
    char name[64],assetid[64];
    int32_t coinid,timestamp,nummsigs,numxps,numwithdraws,pollseconds;
    int64_t txfee,markeramount,NXTheight;
    struct balance_info funds;
    struct unspent_info unspent;
   
    pthread_mutex_t msigs_mutex,xp_mutex;
   
    struct crosschain_info **xps;
   
    struct hashtable *coin_txids,*redeemtxids;
    struct pingpong_queue retryassetxfer,withdrawreqs,withdraws;
    queue_t errorqueue;
    struct multisig_addr **msigs;
    struct withdraw_info **wps;
};

struct daemon_info  // only valid on nodes with access to actual bitcoind (fork) daemons, near realtime validators
{
    char *name[64],*assetid[64],*internalmarker[64],*serverport[64],*userpass[64];          // prepare for one daemon for multiple coinids
    int32_t min_confirms[64],blockheight[64],numinitlist[64];
    int32_t numretries,mismatch,initdone[64],*initlist[64];
    int32_t (*validate_coinaddr)(struct daemon_info *cp,int32_t coinid,char *coinaddr);
    int32_t (*add_unique_txids)(struct daemon_info *cp,int32_t coinid,int64_t blockheight,int32_t RTmode);
    int64_t (*get_blockheight)(struct daemon_info *cp,int32_t coinid);
    queue_t errorqueue;
    char *(*calc_rawtransaction)(struct daemon_info *cp,int32_t coinid,struct rawtransaction *rp,char *destaddr,int64_t amout);
};

struct multigateway_info // only valid if one of NUM_GATEWAYS, does actual money handling
{
    struct hashtable *genaddr_txids;
    struct pingpong_queue genreqs,genaddr,authenticate,sendcoin;
    queue_t errorqueue;

    int32_t gensocks[NUM_GATEWAYS],wdsocks[NUM_GATEWAYS],backupcount,coinid;
    char gateways[NUM_GATEWAYS][64],walletbackup[256];
    struct multisig_addr *(*gen_depositaddr)(struct daemon_info *cp,int32_t coinid,char *NXTaddr);
    char *(*submit_withdraw)(struct daemon_info *cp,int32_t coinid,struct withdraw_info *wp,struct withdraw_info *otherwp);
   
    pthread_mutex_t consensus_mutex;
    struct withdraw_info withdrawinfos[NUM_GATEWAYS];
};

struct gateway_info
{
    struct replicated_coininfo *replicants[64];
    struct daemon_info *daemons[64];
    struct multigateway_info *multigateways[64];
    int32_t gatewayid,timestamp;
};

Feel free to ask any questions. I dont claim these are perfect,just what evolved out of the last month of coding and recoding

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #83 on: April 07, 2014, 01:44:53 pm »

spooky
After all those changes, after a few silly errors, its running!
I thought it would take a long time to get it working again
OK, going to take a break and then run it through the main use cases.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

kodtycoon

  • Hero Member
  • *****
  • Karma: +43/-18
  • Offline Offline
  • Posts: 916
    • View Profile
Re: Multigateway status reports
« Reply #84 on: April 07, 2014, 01:50:01 pm »

nice work james!
Logged

chanc3r

  • Hero Member
  • *****
  • Karma: +124/-50
  • Offline Offline
  • Posts: 1019
  • NXTInspect
    • View Profile
Re: Multigateway status reports
« Reply #85 on: April 07, 2014, 03:45:59 pm »


If you can read C, then maybe you can do a source review? Its grown to ~5000 lines and its hard to see the obvious bugs with code you've been looking at for a long time.

James

I can do 5k lines, will take some time and need you to be stable in design i guess..
Let me know when you want me to have a read...
I can abstract into a doc at the same time to check my understanding.
Logged
NXT: 29996814460165 (NXT-JTA7-B2QR-8BFC-2V222)
@imrimr @NXTinspect

IveBeenBit

  • Full Member
  • ***
  • Karma: +7/-2
  • Offline Offline
  • Posts: 131
    • View Profile
Re: Multigateway status reports
« Reply #86 on: April 07, 2014, 04:00:05 pm »

I can do 5k lines, will take some time and need you to be stable in design i guess..
Let me know when you want me to have a read...
I can abstract into a doc at the same time to check my understanding.

This would be great! Please make sure to post prominently if you follow through on this!
Logged
I need your suggestions for Nxt Podcast ideas. Click here and speak your mind!

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #87 on: April 07, 2014, 04:01:37 pm »


If you can read C, then maybe you can do a source review? Its grown to ~5000 lines and its hard to see the obvious bugs with code you've been looking at for a long time.

James

I can do 5k lines, will take some time and need you to be stable in design i guess..
Let me know when you want me to have a read...
I can abstract into a doc at the same time to check my understanding.
PM me your email and i will send the NXTprotocol layer. That part has been pretty stable for a while and is the foundation for everything else I am working on.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

chanc3r

  • Hero Member
  • *****
  • Karma: +124/-50
  • Offline Offline
  • Posts: 1019
  • NXTInspect
    • View Profile
Re: Multigateway status reports
« Reply #88 on: April 07, 2014, 04:25:12 pm »

PM me your email and i will send the NXTprotocol layer. That part has been pretty stable for a while and is the foundation for everything else I am working on.

James

done.
Logged
NXT: 29996814460165 (NXT-JTA7-B2QR-8BFC-2V222)
@imrimr @NXTinspect

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #89 on: April 07, 2014, 06:13:02 pm »

I just pushed a source release that integrates the NXTprotocol into libwebsockets to https://github.com/jl777/NXTprotocol

NXTprotocol is mostly self-contained, but it does use the following: libcurl, zlib, nacl http://nacl.cr.yp.to/install.html and of course libwebsockets: http://libwebsockets.org/trac/libwebsockets

On ubuntu -lz -lcurl nacl/randombytes.o nacl/libnacl.a will get it to link

On a mac, you need to modify the link.txt file in libwebsockets/lib/Cmakefiles/test-server.dir and add libcurl.dylib nacl/randombytes.o nacl/libnacl.a

You might have to install nacl and generate all the nacl files, only a few are actually used (as of now)

Change the following line in libwebsocketsglue.h to a path on your computer NXTPROTOCOL_HTMLFILE "/Users/jl777/NXTprotocol.html" The NXTprotocol.html file is what is displayed when http://127.0.0.1:7777/ is accessed. It is currently very primitive but shows how to interface to the websockets.

You also need to copy the test-server.c file into the actual libwebsockets/test-server directory

If all goes well, it should just build. There is a stub handler that shows how to hookup a simple NXT service. You can register a specific 4 byte AM signature and then just see those AM's. You can also specify specific transaction type (and subtypes) to filter. You get notified whenever a new block is forged and also polled during idletime. I havent totally verified that it is tracking all the assets properly, especially with all the recent changes, but it does put everything into hashtables or linear lists in RAM.

Anyway, building on this framework, I am able to make a brand new NXT service pretty quickly. I hope it helps you too.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

Daedelus

  • Hero Member
  • *****
  • Karma: +230/-12
  • Offline Offline
  • Posts: 3280
    • View Profile
Re: Multigateway status reports
« Reply #90 on: April 07, 2014, 06:42:17 pm »

What does it actually do?  ???    ;D ;D ;D ;D

I am not having a dig, I just don't understand a lot of your lingo :D  ;D ;D

I want to put it in the Glossary, this is all I can fathom and remember  ;)


****
NXTProtocol
NXTProtocol was written by forum member jl777 and is the foundation that  jl777 built all his NXT Services on top of. These include NxtCash, NxtMixer and others. NxtProtocol simplifies the basic work required to create Services for Nxt and allows jl777, and indeed any other developer, to produce other services on top on the Nxt Platform much easily.

****

I think that is right, is that accurate?

Do you prefer jl777 or James? Could you fill in 'the others' above? I know there are more  ;)

Thanks, I know you are busy  ;D



(Also, imagine you are speaking to a 12 year old in your reply)
Logged
NXT: NXT-4CS7-S4N5-PTH5-A8R2Q

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #91 on: April 07, 2014, 06:50:45 pm »

jl777 is better
This release is primarily so people can get a start on the source code review and wesley can integrate it into  Windows libwebsockets

I included a sample stub handler that shows how you can connect into the NXTprotocol

So, this release is really for developers and not for general use. As soon as we can get it working with windows libwebsockets, then I can think about making a release of something that actually does something.

James

Edit: Chancr said he would document things, I am still pretty busy coding
« Last Edit: April 07, 2014, 06:57:36 pm by farl4bit, Reason: Deleted unnecessary direct full-quote »
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

allwelder

  • Hero Member
  • *****
  • Karma: +196/-13
  • Offline Offline
  • Posts: 1867
  • NxtChina.org
    • View Profile
    • NxtChina.org
Re: Multigateway status reports
« Reply #92 on: April 08, 2014, 12:18:00 am »

JL777 IS MUCH BETTER :D
Logged
NxtChina |Weibo |Twitter Donation welcomed:NXT-APL9-66GU-K8LY-B3JJJ

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #93 on: April 09, 2014, 07:53:07 am »

Some real world stuff coming up, so probably will be offline for a couple days.
Hopefully, the testnet wont be wonky when I get back online

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jabo38

  • Sr. Member
  • ****
  • Karma: +40/-38
  • Offline Offline
  • Posts: 381
    • View Profile
Re: Multigateway status reports
« Reply #94 on: April 09, 2014, 01:58:36 pm »

I just read this thread and I feel sooooo  ;D ;D ;D ;D ;D ;D that I can't control it!
Logged
Never Enough Money

lbjnext

  • Jr. Member
  • **
  • Karma: +0/-0
  • Offline Offline
  • Posts: 21
  • NXT-2HL8-HDMU-7DMZ-8ANYR
    • View Profile
Re: Multigateway status reports
« Reply #95 on: April 11, 2014, 01:10:50 am »

jl777 绝顶高手! ;)
Logged
NXT-2HL8-HDMU-7DMZ-8ANYR

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #96 on: April 11, 2014, 04:31:00 pm »

I have the mood for debugging multigateway, but I have no testNXT
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

Fatih87SK

  • Hero Member
  • *****
  • Karma: +127/-36
  • Offline Offline
  • Posts: 2206
    • View Profile
Re: Multigateway status reports
« Reply #97 on: April 11, 2014, 04:33:34 pm »

I have the mood for debugging multigateway, but I have no testNXT

You should be the one who get's the first testNXT IMO. How can we manage this?
Logged

abctc

  • Hero Member
  • *****
  • Karma: +148/-13
  • Offline Offline
  • Posts: 1396
    • View Profile
Re: Multigateway status reports
« Reply #98 on: April 11, 2014, 04:58:38 pm »

I have the mood for debugging multigateway, but I have no testNXT
- 50k testNXTs sent. Please let me know if you need more.
Logged
Welcome to the Nxt generation of crypto!   Magis quam Moneta (More than a Coin)
"Do not worry, it is an attack" (c) Jean-Luc

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #99 on: April 11, 2014, 10:04:38 pm »

Finally made some progress. With all the testnet resets, each cycle I need to reissue all the assets, redo asset numbers, setup all the accounts, get them funds, etc. Then have to wait until it starts generating blocks in a stable basis and of course testNXT.

I generated the following purely via websockets commands

{
   "NXTaddr":   "8989816935121514892",
   "coinaccts":   [{
         "coin":   "DOGE",
         "funds":   {
            "deposits":   "0",
            "transfers":   "0",
            "withdraws":   "0",
            "moneysends":   "0",
            "remaining_transfers":   "0",
            "remaining_moneysends":   "0",
            "balance":   "0"
         },
         "withdrawaddr":   "",
         "enable_2FA":   0,
         "nummsigs":   1,
         "multisigs":   [{
               "addr":   "9zz4q4QBmc23tpVpRtZLv7oGbdEo9ukGLq",
               "gatewayA":   "209.126.71.170",
               "gatewayB":   "209.126.73.156",
               "gatewayC":   "209.126.73.158"
            }],
         "withdraws":   {
      }
      }, {
         "coin":   "LTC",
         "funds":   {
            "deposits":   "0",
            "transfers":   "0",
            "withdraws":   "0",
            "moneysends":   "0",
            "remaining_transfers":   "0",
            "remaining_moneysends":   "0",
            "balance":   "0"
         },
         "withdrawaddr":   "",
         "enable_2FA":   0,
         "nummsigs":   1,
         "multisigs":   [{
               "addr":   "3Jq2z26hYBharE468BZcvMbzv8UbpeBb6w",
               "gatewayA":   "209.126.71.170",
               "gatewayB":   "209.126.73.156",
               "gatewayC":   "209.126.73.158"
            }],
         "withdraws":   {
      }
      }],
   "numassets":   0,
   "numcoinaccts":   2
}

It is just the address generation, but that involves all three servers, AM parsing, JSON parsing, etc, so now I can proceed to debug the rest of the code after the major restructuring I did last week.

If you have a Mac, you will be able to run the first websockets release. It will automatically generate a new account and print out the password and account number. You will have to send this acct some NXT so it can publish AM's

As long as this incarnation of testnet keeps going, I should be able to get a DOGE/LTC/DRK release this weekend. I dont want to enable BTC until we verify proper functioning on the lesser currencies.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #100 on: April 13, 2014, 02:18:53 am »

chanc3r has started the code review and seems to be about 15% through the code. So far, so good. Nothing major.
I am still battling with the testnet to get my transactions confirmed. It looks like I have everything setup, I get a good txid, it even shows up with getTransaction, but it is not actually there?!?

Kind of hard to test when none of the transactions getconfirmed.

With all of the testnet problems and also in preparation for the upcoming scrutiny at chanc3r's level of detail, I did rework the deposit side to clean up the control flow. I also now use the comment field to get a 100% correlation to the asset transfer. I am seeing a properly constructed transferAsset command go out, but since transactions I am manually doing dont work, the transfers arent working either.

If anybody is able to get their transactions confirmed on testnet, please tell me how!

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

btc2nxt

  • Full Member
  • ***
  • Karma: +42/-1
  • Offline Offline
  • Posts: 211
    • View Profile
Re: Multigateway status reports
« Reply #101 on: April 13, 2014, 03:56:04 am »

chanc3r has started the code review and seems to be about 15% through the code. So far, so good. Nothing major.
I am still battling with the testnet to get my transactions confirmed. It looks like I have everything setup, I get a good txid, it even shows up with getTransaction, but it is not actually there?!?

Kind of hard to test when none of the transactions getconfirmed.

With all of the testnet problems and also in preparation for the upcoming scrutiny at chanc3r's level of detail, I did rework the deposit side to clean up the control flow. I also now use the comment field to get a 100% correlation to the asset transfer. I am seeing a properly constructed transferAsset command go out, but since transactions I am manually doing dont work, the transfers arent working either.

If anybody is able to get their transactions confirmed on testnet, please tell me how!

James

testnet is running fine.

if your transactions can not be confirmed, you can delete h2 database then  reconstruct it.
Logged

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #102 on: April 14, 2014, 01:55:24 am »

None of my local copies are getting any confirmations. I had to use a remote server, but that is so slow and still things are a bit wonky. I bypassed internal checks so I can actually get some progress and managed to get deposits in and withdraws to the point where signrawtransaction doesnt work :(

It used to work, not sure what I did wrong. So, to relieve the stress from this, I decided to write Anon's physical goods store today. got almost all of the basics done and released.

I will wait until I can use the GUI to access a stable testnet as I think the multigateway is pretty much working, but I cant verify it. The types of bugs that are left are the ones where I need to run many test cases, which is not practical now.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #103 on: April 15, 2014, 04:49:34 am »

Latest release has removed the strange behavior. Spent the last day on NXTorrent, but it uses the same NXTprotocol/websockets framework, so I ended up getting a forms based local webpage that talks to multigateway.

http://209.126.71.170/api.html

So now I can finally debug multigateway and with an easy to use web form I hope it will go quickly.
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #104 on: April 16, 2014, 01:47:49 am »

Finally!
I have BTC, LTC, DOGE and DRK all running and deposits work for all. I got asset transfers working for BTC and DOGE, and it probably works for the others too, just havent tested it yet.

Withdraw request also works and the last roadblock was that multisig withdrawal stopped working. It turns out bitcoind is pretty braindead when it comes to multisig and it doesnt even seem to know that it exists. However, you can still send it low level commands and if you just happen to have all the vins, vouts, unspents, etc. correct with proper keys (public and private), redeemscripts and scriptPubKey, then the transaction will go through. At least I just got a manual one to work.

So now, I have to tweak my rawtransaction code to compensate and I will be really close to being able to make a release for Mac's

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #105 on: April 16, 2014, 05:52:07 am »

http://209.126.71.170/multigateway_NXTorrent_v02.zip has a combined Mac version of multigateway and NXTorrent. For the brave, the source is at https://github.com/jl777/NXTprotocol/tree/master/multigateway

It really needs a lot of testing. It is currently limited to DOGE, I got exactly 1 multisig withdraw to work, so no guarantees you will be able to withdraw. I am not seeing any problems in getting the deposits, so your DOGEs will be safe.

There is one file in the ZIP file, just run it and it will launch a command line program. First thing it does is generate a high entropy password (along with a bunch of files, so best to put it in its own directory). It will print this address at the top of http://127.0.0.1:7777  You need to send it enough testNXT so it can send AM's

You might have to wait a bit and sometimes it returns blank page. just hit refresh. You can see multigateway stats, filter by NXT acct or coin. Before you can make a deposit, you need to generate one and you can set your withdraw address in the same AM.

I have enabled tokenized security,which means all non-display functions will fail at first. It will display the actual command you need with the proper token calculated. copy it and paste it over starting at the ? in the URL.
127.0.0.1:7777/multigateway?<paste the tokenized JSON here>

For the curious, you can look at the commandline window and see a bunch of status going by.

Any crashes, or problems, definitely post here,along with the last part of the command window. I will probably offline most of the day, so dont worry if not response for a bit. It would be nice to get some test results. I am pretty sure the display of account info is not quite right.

James

P.S. NXTorrent does not have tokenized security (yet) and it could use some testing too
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

Fatih87SK

  • Hero Member
  • *****
  • Karma: +127/-36
  • Offline Offline
  • Posts: 2206
    • View Profile
Re: Multigateway status reports
« Reply #106 on: April 16, 2014, 11:05:28 am »

Hi jl777,

There was an idea to release MultiGateway together with Asset Exchange.

What do you think about the planning of these two features? Will there be a big gap of releasedates or could it be ready at the same time?

If you can answer this we can decide about it.
Logged

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #107 on: April 16, 2014, 02:27:18 pm »

Hi jl777,

There was an idea to release MultiGateway together with Asset Exchange.

What do you think about the planning of these two features? Will there be a big gap of releasedates or could it be ready at the same time?

If you can answer this we can decide about it.
I cant release multigateway before AE as it depends on it
Also, there has been very little testing of multigateway by anybody other than me, so I am not comfortable with its stability yet. source review is proceeding and it could uncover some things that require some time.

Realistically, the guys testing AE need to finish testing AE so we can get that launched. Then hopefully they will be able to validate multigateway for release. It is really a matter of getting the tests, but there seem to be very few people for that...

So, simultaneous release is not practical. There will be a version running on testnet, but we also need to get production servers deployed and hardened. Again this will take some time. The more testers, the faster. Currently there doesnt seem to be anybody testing other than me, so it will be a long time

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #108 on: April 16, 2014, 04:24:31 pm »

Looking into porting multigateway, NXTorrent, etc. to Windows. It seems that pthreads, sockets and time functions are not standard for windows. If anybody knows of a good cross platform equivalent for:

pthreads
sockets
time functions

please let me know

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

Fatih87SK

  • Hero Member
  • *****
  • Karma: +127/-36
  • Offline Offline
  • Posts: 2206
    • View Profile
Re: Multigateway status reports
« Reply #109 on: April 16, 2014, 04:52:21 pm »

Got it. Thanks!
Logged

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #110 on: April 16, 2014, 08:23:09 pm »

chanc3r is getting in to high gear and starting to find some bugs. I didnt state any specify bounty for bugs found, but he has definitely earned it!

For finding memory leaks and a host of small issues, he got 1000 NXT
I am no BCNext, so there will be plenty of bugs for people to find. No guarantees on amounts paid, but the more severe the problem you find, the more you will get.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

emule

  • Jr. Member
  • **
  • Karma: +15/-64
  • Offline Offline
  • Posts: 49
    • View Profile
Re: Multigateway status reports
« Reply #111 on: April 16, 2014, 08:37:32 pm »



is this working yet ? zzzzzzzz :-\
Logged

Daedelus

  • Hero Member
  • *****
  • Karma: +230/-12
  • Offline Offline
  • Posts: 3280
    • View Profile
Re: Multigateway status reports
« Reply #112 on: April 16, 2014, 11:54:12 pm »

James,

I take it with all you projects being public that none of them are 'commercially sensitive'?

Its all open source yadda yadda yadda?   ;D
Logged
NXT: NXT-4CS7-S4N5-PTH5-A8R2Q

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #113 on: April 17, 2014, 12:43:31 am »

James,

I take it with all you projects being public that none of them are 'commercially sensitive'?

Its all open source yadda yadda yadda?   ;D
I dont fully understand the question...
I have used a few external source like cJSON, nacl, zlib, but they are all open source
Most of my projects are from code that I write.

https://github.com/jl777/NXTprotocol

I intend to make all my projects open source, usually around the time it goes into beta test. That way, source review ties in with the debugging process

Does that answer your question?

James

P.S. I sometimes do consulting work and deliver closed source projects, but that is for private businesses
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

Daedelus

  • Hero Member
  • *****
  • Karma: +230/-12
  • Offline Offline
  • Posts: 3280
    • View Profile
Re: Multigateway status reports
« Reply #114 on: April 17, 2014, 01:37:00 am »

James,

I take it with all you projects being public that none of them are 'commercially sensitive'?

Its all open source yadda yadda yadda?   ;D
I dont fully understand the question...
I have used a few external source like cJSON, nacl, zlib, but they are all open source
Most of my projects are from code that I write.

https://github.com/jl777/NXTprotocol

I intend to make all my projects open source, usually around the time it goes into beta test. That way, source review ties in with the debugging process

Does that answer your question?

James

P.S. I sometimes do consulting work and deliver closed source projects, but that is for private businesses

Sorry, I was very unclear  ;D

Explicitly, is there enough info on this thread to allow someone to poach and implement multisig, possibly before you do?

I have been spreading the word about multisig gateways and ripple fiat bridges and pointing them to this thread via pm. I was worried this wasn't a good idea :-[ should I still do this? There is a lot of interest.
Logged
NXT: NXT-4CS7-S4N5-PTH5-A8R2Q

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #115 on: April 17, 2014, 02:00:58 am »

James,

I take it with all you projects being public that none of them are 'commercially sensitive'?

Its all open source yadda yadda yadda?   ;D
I dont fully understand the question...
I have used a few external source like cJSON, nacl, zlib, but they are all open source
Most of my projects are from code that I write.

https://github.com/jl777/NXTprotocol

I intend to make all my projects open source, usually around the time it goes into beta test. That way, source review ties in with the debugging process

Does that answer your question?

James

P.S. I sometimes do consulting work and deliver closed source projects, but that is for private businesses

Sorry, I was very unclear  ;D

Explicitly, is there enough info on this thread to allow someone to poach and implement multisig, possibly before you do?

I have been spreading the word about multisig gateways and ripple fiat bridges and pointing them to this thread via pm. I was worried this wasn't a good idea :-[ should I still do this? There is a lot of interest.
Are you talking about a multisig gateway for non-NXT platforms or for NXT platforms. If the former, it would require a fair amount of reworking as NXT-ness is embedded throughout.

I will make sure to keep the "secret sauce" internal, at least for a little bit. I will share with trusted members, so we can get a full source review

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

Daedelus

  • Hero Member
  • *****
  • Karma: +230/-12
  • Offline Offline
  • Posts: 3280
    • View Profile
Re: Multigateway status reports
« Reply #116 on: April 17, 2014, 02:20:25 am »

Great, as long as the sauce is safe and copying hard I will keep spreading the word ;D
Logged
NXT: NXT-4CS7-S4N5-PTH5-A8R2Q

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #117 on: April 18, 2014, 02:05:51 am »

I wasnt feeling too well today, so I didnt want to mess with multigateway. I ended up with a crazy idea that maybe just means I am really sick and delusional, or I have come up with a way to implement atomic cross chain exchanges!

Before getting too excited, I need a volunteer who has a Mac. I want to test LTC <-> DOGE atomic exchanges, so you need to have both litecoin and dogecoin wallets installed with the following in the conf files:

/Users/<your username>/Library/Application Support/Dogecoin/dogecoin.conf
/Users/<your username>/Library/Application Support/Litecoin/litecoin.conf

server=1
rpcuser=dogecoinrpc
rpcpassword=<wallet will tell you a password to use>
txindex=1

It will be totally user unfriendly to setup and you will have to manually input the trade parameters. I just want to verify that this method really works.

Since its not implemented as the usual atomic methods (from what I can tell, the current miners wont even relay those tx's), I am calling this "NXTsubatomic" exchange. If I am right, it will work with the current miners for most all the bitcoind forks.

PM me if you are willing to be a guinea pig

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #118 on: April 18, 2014, 06:28:50 am »

I did manage to get the libwebsockets working on ubuntu, at least I think it is working. I only have remote server access and no desktop.

http://209.126.71.170/ubuntu.zip has a test file. I dont expect it to do much, just hope that it interacts with http://209.126.71.170/api.html

Not all forms work, but the NXTorrent section should at least do something

If you have ubuntu desktop, just download the ubuntu.zip and run the NXTprotocol_test file.

I would like to know if you get anything interesting happening in the text window or the browser

VanBreuk pointed me to a cross platform sockets library and I got that integrated in for ubuntu and Mac. In the laborious process of getting a Windows dev environment setup. Theoretically everything should be portable as there is even a pthread library for windows.

Making slow progress in getting to a cross platform code base. Not sure how long this will take, but it seems clear that it needs to be done.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

fanaticalfishing

  • Full Member
  • ***
  • Karma: +10/-1
  • Offline Offline
  • Posts: 149
    • View Profile
Re: Multigateway status reports
« Reply #119 on: April 18, 2014, 05:11:28 pm »

I did manage to get the libwebsockets working on ubuntu, at least I think it is working. I only have remote server access and no desktop.

http://209.126.71.170/ubuntu.zip has a test file. I dont expect it to do much, just hope that it interacts with http://209.126.71.170/api.html

Not all forms work, but the NXTorrent section should at least do something

If you have ubuntu desktop, just download the ubuntu.zip and run the NXTprotocol_test file.

I would like to know if you get anything interesting happening in the text window or the browser

VanBreuk pointed me to a cross platform sockets library and I got that integrated in for ubuntu and Mac. In the laborious process of getting a Windows dev environment setup. Theoretically everything should be portable as there is even a pthread library for windows.

Making slow progress in getting to a cross platform code base. Not sure how long this will take, but it seems clear that it needs to be done.

James

it seems to be doing something on ubuntu.
The console shows it is scanning blocks.

Code: [Select]
NXT.74329 HIST timestamp.12545905 lag.497 NEW block.(884148407653481877) vs lastblock.(5220721533726672280) -> (5757168833205299378)
NXT.74330 HIST timestamp.12545982 lag.420 NEW block.(5757168833205299378) vs lastblock.(884148407653481877) -> (12746900890174342166)
HIST Waiting for hashtable queue to catchup: 1397840801
HIST Historical processing has finished: 1397840801
NXT.74333  timestamp.12546415 lag.11 NEW block.(7198590531545550036) vs lastblock.(16562822566049108305) -> ()
NXT.74334  timestamp.12546554 lag.32 NEW block.(13276547175619594509) vs lastblock.(7198590531545550036) -> (5006581464986821178)
NXT.74335  timestamp.12546565 lag.31 NEW block.(5006581464986821178) vs lastblock.(13276547175619594509) -> ()
NXT.74336  timestamp.12546589 lag.17 NEW block.(15215150350929899546) vs lastblock.(5006581464986821178) -> ()

I dont see anything different in the web browser. Im using nrs 9.6

Logged

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #120 on: April 18, 2014, 05:28:25 pm »

I did manage to get the libwebsockets working on ubuntu, at least I think it is working. I only have remote server access and no desktop.

http://209.126.71.170/ubuntu.zip has a test file. I dont expect it to do much, just hope that it interacts with http://209.126.71.170/api.html

Not all forms work, but the NXTorrent section should at least do something

If you have ubuntu desktop, just download the ubuntu.zip and run the NXTprotocol_test file.

I would like to know if you get anything interesting happening in the text window or the browser

VanBreuk pointed me to a cross platform sockets library and I got that integrated in for ubuntu and Mac. In the laborious process of getting a Windows dev environment setup. Theoretically everything should be portable as there is even a pthread library for windows.

Making slow progress in getting to a cross platform code base. Not sure how long this will take, but it seems clear that it needs to be done.

James

it seems to be doing something on ubuntu.
The console shows it is scanning blocks.

Code: [Select]
NXT.74329 HIST timestamp.12545905 lag.497 NEW block.(884148407653481877) vs lastblock.(5220721533726672280) -> (5757168833205299378)
NXT.74330 HIST timestamp.12545982 lag.420 NEW block.(5757168833205299378) vs lastblock.(884148407653481877) -> (12746900890174342166)
HIST Waiting for hashtable queue to catchup: 1397840801
HIST Historical processing has finished: 1397840801
NXT.74333  timestamp.12546415 lag.11 NEW block.(7198590531545550036) vs lastblock.(16562822566049108305) -> ()
NXT.74334  timestamp.12546554 lag.32 NEW block.(13276547175619594509) vs lastblock.(7198590531545550036) -> (5006581464986821178)
NXT.74335  timestamp.12546565 lag.31 NEW block.(5006581464986821178) vs lastblock.(13276547175619594509) -> ()
NXT.74336  timestamp.12546589 lag.17 NEW block.(15215150350929899546) vs lastblock.(5006581464986821178) -> ()

I dont see anything different in the web browser. Im using nrs 9.6
if you go to the api.html page and submit a NXTorrent listing, what appears in the text window?
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

fanaticalfishing

  • Full Member
  • ***
  • Karma: +10/-1
  • Offline Offline
  • Posts: 149
    • View Profile
Re: Multigateway status reports
« Reply #121 on: April 18, 2014, 06:56:24 pm »

I did manage to get the libwebsockets working on ubuntu, at least I think it is working. I only have remote server access and no desktop.

http://209.126.71.170/ubuntu.zip has a test file. I dont expect it to do much, just hope that it interacts with http://209.126.71.170/api.html

Not all forms work, but the NXTorrent section should at least do something

If you have ubuntu desktop, just download the ubuntu.zip and run the NXTprotocol_test file.

I would like to know if you get anything interesting happening in the text window or the browser

VanBreuk pointed me to a cross platform sockets library and I got that integrated in for ubuntu and Mac. In the laborious process of getting a Windows dev environment setup. Theoretically everything should be portable as there is even a pthread library for windows.

Making slow progress in getting to a cross platform code base. Not sure how long this will take, but it seems clear that it needs to be done.

James

it seems to be doing something on ubuntu.
The console shows it is scanning blocks.

Code: [Select]
NXT.74329 HIST timestamp.12545905 lag.497 NEW block.(884148407653481877) vs lastblock.(5220721533726672280) -> (5757168833205299378)
NXT.74330 HIST timestamp.12545982 lag.420 NEW block.(5757168833205299378) vs lastblock.(884148407653481877) -> (12746900890174342166)
HIST Waiting for hashtable queue to catchup: 1397840801
HIST Historical processing has finished: 1397840801
NXT.74333  timestamp.12546415 lag.11 NEW block.(7198590531545550036) vs lastblock.(16562822566049108305) -> ()
NXT.74334  timestamp.12546554 lag.32 NEW block.(13276547175619594509) vs lastblock.(7198590531545550036) -> (5006581464986821178)
NXT.74335  timestamp.12546565 lag.31 NEW block.(5006581464986821178) vs lastblock.(13276547175619594509) -> ()
NXT.74336  timestamp.12546589 lag.17 NEW block.(15215150350929899546) vs lastblock.(5006581464986821178) -> ()

I dont see anything different in the web browser. Im using nrs 9.6
if you go to the api.html page and submit a NXTorrent listing, what appears in the text window?

When accessing http://127.0.0.1:7777/api with firefox
Code: [Select]
    GET URI = /api n.0
    Host = 127.0.0.1:7777 n.2
    Connection = keep-alive n.3
    Accept: = text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 n.18
    Accept-Encoding: = gzip, deflate n.20
    Accept-Language: = en-US,en;q=0.5 n.21
lwsts[2454]: Unknown mimetype for /usr/local/share/api

This is at the bottom of the api page
Quote
HTTP/1.0 415 Unsupported Media Type Server: libwebsockets Content-Type: text/html
415 Unsupported Media Type


When posting this listing:

Post an Item for Sale:
Item Title:    Tribbles
Item Type:    Pets
Item Description:    Fuzzy Furry Tribbles
Item Price (in NXT):    25

I see no reponse in the terminal

Logged

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #122 on: April 18, 2014, 07:10:35 pm »

I did manage to get the libwebsockets working on ubuntu, at least I think it is working. I only have remote server access and no desktop.

http://209.126.71.170/ubuntu.zip has a test file. I dont expect it to do much, just hope that it interacts with http://209.126.71.170/api.html

Not all forms work, but the NXTorrent section should at least do something

If you have ubuntu desktop, just download the ubuntu.zip and run the NXTprotocol_test file.

I would like to know if you get anything interesting happening in the text window or the browser

VanBreuk pointed me to a cross platform sockets library and I got that integrated in for ubuntu and Mac. In the laborious process of getting a Windows dev environment setup. Theoretically everything should be portable as there is even a pthread library for windows.

Making slow progress in getting to a cross platform code base. Not sure how long this will take, but it seems clear that it needs to be done.

James

it seems to be doing something on ubuntu.
The console shows it is scanning blocks.

Code: [Select]
NXT.74329 HIST timestamp.12545905 lag.497 NEW block.(884148407653481877) vs lastblock.(5220721533726672280) -> (5757168833205299378)
NXT.74330 HIST timestamp.12545982 lag.420 NEW block.(5757168833205299378) vs lastblock.(884148407653481877) -> (12746900890174342166)
HIST Waiting for hashtable queue to catchup: 1397840801
HIST Historical processing has finished: 1397840801
NXT.74333  timestamp.12546415 lag.11 NEW block.(7198590531545550036) vs lastblock.(16562822566049108305) -> ()
NXT.74334  timestamp.12546554 lag.32 NEW block.(13276547175619594509) vs lastblock.(7198590531545550036) -> (5006581464986821178)
NXT.74335  timestamp.12546565 lag.31 NEW block.(5006581464986821178) vs lastblock.(13276547175619594509) -> ()
NXT.74336  timestamp.12546589 lag.17 NEW block.(15215150350929899546) vs lastblock.(5006581464986821178) -> ()

I dont see anything different in the web browser. Im using nrs 9.6
if you go to the api.html page and submit a NXTorrent listing, what appears in the text window?

When accessing http://127.0.0.1:7777/api with firefox
Code: [Select]
    GET URI = /api n.0
    Host = 127.0.0.1:7777 n.2
    Connection = keep-alive n.3
    Accept: = text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 n.18
    Accept-Encoding: = gzip, deflate n.20
    Accept-Language: = en-US,en;q=0.5 n.21
lwsts[2454]: Unknown mimetype for /usr/local/share/api

This is at the bottom of the api page
Quote
HTTP/1.0 415 Unsupported Media Type Server: libwebsockets Content-Type: text/html
415 Unsupported Media Type


When posting this listing:

Post an Item for Sale:
Item Title:    Tribbles
Item Type:    Pets
Item Description:    Fuzzy Furry Tribbles
Item Price (in NXT):    25

I see no reponse in the terminal
I am having some problems with the forms. I think it is just a matter of getting the HTML handshake fixed.
At least it is running on ubuntu. I will look into this next week
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

kodtycoon

  • Hero Member
  • *****
  • Karma: +43/-18
  • Offline Offline
  • Posts: 916
    • View Profile
Re: Multigateway status reports
« Reply #123 on: April 18, 2014, 07:33:39 pm »

saturn coin going pos and multipool just announced buy orders plowed throught 15btc worth of sell orders at 8 9 10 satashis not at 11.. just letting you know...
Logged

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #124 on: April 21, 2014, 08:16:11 pm »

http://www.righto.com/2014/02/bitcoins-hard-way-using-raw-bitcoin.html
http://exiledbear.wordpress.com/2013/06/06/criticisms-of-bitcoins-raw-txn-format/

NXT core is so much cleaner! I'm not the only one puzzled by bitcoin's tx format, then again I havent delved into the NXT raw format,but so far havent had to

Anyway, just wanted to let people know that multigateway is being code reviewed and I am also preparing Windows compatibility for the websockets exe that will include all my projects in a single cross platform release. I will be using libuv which is used by some other well known projects, like node.js

A bit of a delay, but it will be well worth it to get a scalable platform, especially since libuv supports plugins, threading and networking in a cross platform way

James

P.S. I am also figuring out how to properly integrate multigateway with NXTsubatomic (and NXTatomic) and InstantDEX.
« Last Edit: April 21, 2014, 08:44:44 pm by jl777 »
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

Daedelus

  • Hero Member
  • *****
  • Karma: +230/-12
  • Offline Offline
  • Posts: 3280
    • View Profile
Re: Multigateway status reports
« Reply #125 on: April 21, 2014, 09:28:00 pm »

Will it be easy to add blackcoin?

The reason I ask that it is the first mining community that see the benefits of full POS. They like the multipool idea as they can keep mining. Know any other POS 'coins' they might be interested in? ;D

Or you can really pee them off by adding whitecoin, which I think is a fork of blackcoin :D ;D
« Last Edit: April 21, 2014, 09:45:02 pm by Daedelus »
Logged
NXT: NXT-4CS7-S4N5-PTH5-A8R2Q

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #126 on: April 21, 2014, 09:56:27 pm »

Will it be easy to add blackcoin?

The reason I ask that it is the first mining community that see the benefits of full POS. They like the multipool idea as they can keep mining. Know any other POS 'coins' they might be interested in? ;D
I added it to the source code and am downloading the blockchain. So yes, I can add it, but since it is a peercoin clone and peercoin doesnt support multisig yet, it probably wont work...
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

Daedelus

  • Hero Member
  • *****
  • Karma: +230/-12
  • Offline Offline
  • Posts: 3280
    • View Profile
Re: Multigateway status reports
« Reply #127 on: April 21, 2014, 10:09:13 pm »

Will it be easy to add blackcoin?

The reason I ask that it is the first mining community that see the benefits of full POS. They like the multipool idea as they can keep mining. Know any other POS 'coins' they might be interested in? ;D
I added it to the source code and am downloading the block chain. So yes, I can add it, but since it is a peercoin clone and peercoin doesnt support multisig yet, it probably wont work...

Oh yes, I remember you mentioned that before :-\

They are big on the multisig idea when i bought it up but onlyevery other coin ever will be too. So no biggy ;D
Logged
NXT: NXT-4CS7-S4N5-PTH5-A8R2Q

Jimmy2011

  • Sr. Member
  • ****
  • Karma: +24/-19
  • Offline Offline
  • Posts: 329
    • View Profile
Re: Multigateway status reports
« Reply #128 on: April 22, 2014, 02:29:48 pm »


As seen from peercointalk.org that the Peercoin community is working on the multiSig problem. Hope progress in the near future.
Logged
NXT-LX5G-L63N-ST8S-9LVZY

Daedelus

  • Hero Member
  • *****
  • Karma: +230/-12
  • Offline Offline
  • Posts: 3280
    • View Profile
Re: Multigateway status reports
« Reply #129 on: April 22, 2014, 02:42:06 pm »

link pls?

Thanks  ;D
Logged
NXT: NXT-4CS7-S4N5-PTH5-A8R2Q

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #130 on: April 22, 2014, 10:09:35 pm »

Just released NXTsubatomic_v00 for testing, but since testnet is still wonky, I think I will see if I can get an atomic swap of two NXT transactions.

That will sure be nice and handy to be able to do!

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

Daedelus

  • Hero Member
  • *****
  • Karma: +230/-12
  • Offline Offline
  • Posts: 3280
    • View Profile
Re: Multigateway status reports
« Reply #131 on: April 22, 2014, 10:52:48 pm »

Jame,

I have a feeling you answered this somewhere already but the guys are asking about a multisig and ae tie up.

See https://nxtforum.org/nxt-promotion/promotion-of-ae/msg10906/#new
Logged
NXT: NXT-4CS7-S4N5-PTH5-A8R2Q

Daedelus

  • Hero Member
  • *****
  • Karma: +230/-12
  • Offline Offline
  • Posts: 3280
    • View Profile
Re: Multigateway status reports
« Reply #132 on: April 30, 2014, 01:50:57 pm »

You have another fan asking when multisig will be ready  ::)   ;D

I have posted an extract from the comments you made of the 22nd April, with a link. Update them if you want  ;D ;D ;D

https://bitcointalk.org/index.php?topic=587007.msg6474122;topicseen#new
Logged
NXT: NXT-4CS7-S4N5-PTH5-A8R2Q

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #133 on: May 12, 2014, 12:09:49 am »

For those who missed it, I made a release of NXTservices_v03, which includes deposit support for multigateway.
Working on porting withdraw support to the new codebase today.
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #134 on: May 12, 2014, 04:25:17 am »

All the inconsistent curl behavior bugged me to no end. So I rewrote the bitcoind_RPC.c and now the Mac which used to go really fast all the time, now gets sporadic 2 second pauses. Really kills performance.

Anyway, I finally found a pretty good article that describes the problem http://devblog.songkick.com/2012/11/27/a-second-here-a-second-there/

So, now mac is back to full speed. Now lets hope the ubuntu and Windows performance is also fixed.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

fanaticalfishing

  • Full Member
  • ***
  • Karma: +10/-1
  • Offline Offline
  • Posts: 149
    • View Profile
Re: Multigateway status reports
« Reply #135 on: May 12, 2014, 03:37:13 pm »

I am seeing a lot of errors with the ubuntu client v.0

Code: [Select]
0x1d74600 key.() len.0 is too big for NXTaddrs 24, FATAL
0x7f74d8df3e90 key.() len.0 is too big for NXTaddrs 24, FATAL
0x1d74600 key.() len.0 is too big for NXTaddrs 24, FATAL
0x7f74d8df3e90 key.() len.0 is too big for NXTaddrs 24, FATAL
0x1d74600 key.() len.0 is too big for NXTaddrs 24, FATAL
0x7f74d8df3e90 key.() len.0 is too big for NXTaddrs 24, FATAL
0x1d74600 key.() len.0 is too big for NXTaddrs 24, FATAL
0x7f74d8df3e90 key.() len.0 is too big for NXTaddrs 24, FATAL
0x1d74600 key.() len.0 is too big for NXTaddrs 24, FATAL
0x7f74d8df3e90 key.() len.0 is too big for NXTaddrs 24, FATAL
NXT.95437  timestamp.14613801 lag.-15 NEW block.(16484442309354311875) vs lastblock.(13054178980505095375) -> ()
send 600 bytes to socket.14
Logged

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #136 on: May 12, 2014, 03:42:23 pm »

I am seeing a lot of errors with the ubuntu client v.0

Code: [Select]
0x1d74600 key.() len.0 is too big for NXTaddrs 24, FATAL
0x7f74d8df3e90 key.() len.0 is too big for NXTaddrs 24, FATAL
0x1d74600 key.() len.0 is too big for NXTaddrs 24, FATAL
0x7f74d8df3e90 key.() len.0 is too big for NXTaddrs 24, FATAL
0x1d74600 key.() len.0 is too big for NXTaddrs 24, FATAL
0x7f74d8df3e90 key.() len.0 is too big for NXTaddrs 24, FATAL
0x1d74600 key.() len.0 is too big for NXTaddrs 24, FATAL
0x7f74d8df3e90 key.() len.0 is too big for NXTaddrs 24, FATAL
0x1d74600 key.() len.0 is too big for NXTaddrs 24, FATAL
0x7f74d8df3e90 key.() len.0 is too big for NXTaddrs 24, FATAL
NXT.95437  timestamp.14613801 lag.-15 NEW block.(16484442309354311875) vs lastblock.(13054178980505095375) -> ()
send 600 bytes to socket.14
Those error messages indicate your didnt disable POST enforcement in the .conf file
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

fanaticalfishing

  • Full Member
  • ***
  • Karma: +10/-1
  • Offline Offline
  • Posts: 149
    • View Profile
Re: Multigateway status reports
« Reply #137 on: May 12, 2014, 06:04:06 pm »



Is this discussed somewhere? If so I missed it.

I do not see a .conf file. Is in the ~/ directory or should it be in the application folder?

Logged

PoofKnuckle

  • Hero Member
  • *****
  • Karma: +111/-7
  • Offline Offline
  • Posts: 849
  • Your Concern is Noted.
    • View Profile
    • NXT FreeMarket
Re: Multigateway status reports
« Reply #138 on: May 12, 2014, 06:07:04 pm »



Is this discussed somewhere? If so I missed it.

I do not see a .conf file. Is in the ~/ directory or should it be in the application folder?

That would be the .conf file in your NXT folder. Looking at my machine, it's:

nxt/conf/nxt-default.properties

So, not actually a .conf file :)

Logged
NXT: NXT-WBVP-83YM-FRB3-9MQLY
BM-NBTVHaA41Ejh2G6krD9p5vM62vk84isM

fanaticalfishing

  • Full Member
  • ***
  • Karma: +10/-1
  • Offline Offline
  • Posts: 149
    • View Profile
Re: Multigateway status reports
« Reply #139 on: May 12, 2014, 06:40:53 pm »



Is this discussed somewhere? If so I missed it.

I do not see a .conf file. Is in the ~/ directory or should it be in the application folder?

That would be the .conf file in your NXT folder. Looking at my machine, it's:

nxt/conf/nxt-default.properties

So, not actually a .conf file :)

I SEE! I SEE! Said the blind man. :-)
It is working now, many thanks.
Logged

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #140 on: May 13, 2014, 10:31:04 am »

after 4hrs sleep after all nighter due to AE launch I wasnt able to code today,but I did make a breakthrough on the InstantDEX design. now even if evilbob forges the next block and is doing a direct tx with you, the worst he can do is not do the trade he agreed to. considering how rare it will be and it is just an annoyance without any funds lost, I consider this attack vector dealt with!

Have a few pretty major things cooking, but not ready to announce anything yet. very exciting times for NXT!!!

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

Daedelus

  • Hero Member
  • *****
  • Karma: +230/-12
  • Offline Offline
  • Posts: 3280
    • View Profile
Re: Multigateway status reports
« Reply #141 on: May 13, 2014, 10:53:27 am »

Are these announcements of things brand new porjects or have you trailed them a little already?  ;D
Logged
NXT: NXT-4CS7-S4N5-PTH5-A8R2Q

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #142 on: May 13, 2014, 11:05:24 am »

Are these announcements of things brand new porjects or have you trailed them a little already?  ;D
I am working on three deals. One tech, one finance, and one that is neither. all relate to NXT of course
This is what happens when I dont get enough sleep. I cant code, but I have to do something :)

Now that things have calmed down I will be going back into coding mode. I had been providing personal inventory for both InstantDEX and NXTcoinsco, but after what little is still on the books are gone, it will be up to the early purchasers to provide some inventory. I have found that I cant code and trade at the same time, at least not effectively as it seems to use very similar parts of my brain.

James

P.S. I have put small buy walls to provide a trading range for traders. Those I dont have to think about so they will stay.
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

swartzfeger

  • Hero Member
  • *****
  • Karma: +50/-1
  • Offline Offline
  • Posts: 611
  • I bent my wookie
    • View Profile
    • https://www.instagram.com/swartzfeger/
Re: Multigateway status reports
« Reply #143 on: May 14, 2014, 11:35:28 pm »

James, I've been remiss in testing because my smarter partner (chanc3r) has been unavailable due to work. Is there anything I can help test, break, etc?
Logged

CoinTropololis_JustaBit

  • Hero Member
  • *****
  • Karma: +144/-11
  • Offline Offline
  • Posts: 727
    • View Profile
Re: Multigateway status reports
« Reply #144 on: May 14, 2014, 11:47:54 pm »

Exciting times indeed. ;)

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #145 on: May 15, 2014, 01:20:43 am »

James, I've been remiss in testing because my smarter partner (chanc3r) has been unavailable due to work. Is there anything I can help test, break, etc?
I really would like to see people other than me prove that they can chat directly
so just install v03 and establish chat with anybody else that would help
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #146 on: May 16, 2014, 06:37:42 am »

You might have noticed I have been doing a lot of financing recently. The reason for this is that the bitcoind via libcurl is just sooooo slow. It take 20 minutes to an hour to reload and everyday it gets bigger and slower.
With NXT, I can reprocess the whole blockchain in a few minutes. So during this eternal wait, I check forum, PM's, an idea forms and next thing you know there is a new asset.

I think I need to speed up this MGW initialization. It shouldnt be restarted that often once it is deployed but it just isnt practical to debug now. To date, all MGW state info was in the blockchains. this is nice and decentralized but requires a full blockchain scan each time. Too slow.

so, I will code up a way to massively speed this part up to allow rapid cycle debugging. This will take a bit of time. More when I know how much time.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #147 on: May 18, 2014, 01:19:14 am »

I have proof libcurl is the devil!
I tried all sorts of arbitrary values to delay between calls to prevent from going mysteriously non-responsive for many seconds.

Of course when i used 666 milliseconds, the errors go away.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

psic4t

  • Jr. Member
  • **
  • Karma: +4/-0
  • Offline Offline
  • Posts: 55
  • NXT-NHPU-6WBD-CPNG-2CXCE
    • View Profile
Re: Multigateway status reports
« Reply #148 on: May 18, 2014, 01:39:58 am »

Of course when i used 666 milliseconds, the errors go away.

Hahaha. Beat it!
Logged

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #149 on: May 21, 2014, 08:34:22 pm »

Finally, now that AE launch is a week behind us, I able to finish all the nontech work with more than a few hours left in the day!

That means I can actually write code again.

So back to the MGW issue that I have to put silly pauses before each RPC command to reduce the number of 10 seconds sleeps that seems to be the only way to get things working again. OK this doesnt sound that bad until you count the number of RPC calls are needed to rescan the BTC blockchain. Ever wonder why reprocessing the blockchain takes so long?

The reason is that it does not seem to be optimized at all. They assumed that the network will be the bottleneck and did not count on tens of thousands of requests per second that my code issues. I use hashtables that have on average two iterations regardless of table entries so it is effectively constant time searching. Basically everything is really fast, except for the libcurl.

Each BTC block has a lot of transactions, so reprocessing requires 1 RPC to get the block and N*2 RPC for each tx in the block since I have to getrawtx and decode it. bitcoind does not properly associate multisig addrs to accts. It really has been a challenge to make MGW work properly

Anyway the practical problem was that each time I made a new version I had to wait for 30+ minutes to reprocess. To me that is a LONG time. So I check PM's, catchup on forum and next thing you know I am far away from debugging. By the time I notice that it has finally reprocessed I usually cant remember what change I just made.

Something had to be done. I like interactive debugging. Make a small change, verify it works, repeat until all bugs are fixed. I make hundreds of test versions per day when actively debugging.

Anyway, I was considering gutting the current control flow to deal with this issue, but I just couldnt get myself to do it. Afterall, it is 99% working and any new implementation has no guarantee of being that much better. But to speed it up requires a massive overall. Or does it?

Dont know why I didnt think of it before, lack of sleep and time is what I blame :)
Good news is that I am current waiting for the test of the sped up version. Still not great, but instead of 30+ minutes, less than 5 minutes to reprocess!

So, I can finally semi-interactively debug MGW

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #150 on: May 22, 2014, 12:08:08 am »

Must faster now, processing around 100000 requests per second. Still not optimally fast, but certainly much better than before, about 100 times faster!

Now I can run all four daemons and get a reasonable debug cycle, but it will take an hour or so to initialize the first time.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

ebby

  • Full Member
  • ***
  • Karma: +21/-2
  • Offline Offline
  • Posts: 190
    • View Profile
Re: Multigateway status reports
« Reply #151 on: May 23, 2014, 05:14:52 pm »

Nice to see some success again with MGW, thanks jl777!

Any eta when MGW will be ready for Mainnet? I think everyone is waiting for it.
Logged
Tips and donations are much appreciated.
NXT: NXT-HPW9-EC4H-KJPT-7STQV (Hallmarked 100% solar powered cubietruck node (uptime 107 days!). If this is not Green, what else?)

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #152 on: May 23, 2014, 06:03:15 pm »

Nice to see some success again with MGW, thanks jl777!

Any eta when MGW will be ready for Mainnet? I think everyone is waiting for it.
I am struggling with the current method of tracking account balances. It works, but it is too complicated for me to make changes to it easily.
This is usually a sign that it need to be reworked.

Now I am very reluctant to make wholesale changes at this point, but the fact that after two days of trying to make a "simple" change (the three MGW servers dont agree on which withdrawal to do next and deadlocks) I am at the point of biting the bullet and reworking it.

So, I need some help and other eyes to make sure that after this reworking MGW balances will be totally accurate.

Now that I sped up reprocessing 100 times, I can consider more brute force methods as opposed to the current state machine method. As it is, any error anywhere, ever, would propagate forever. While my code has very low error rates, the forever propagation concerns me, especially if a HDD hiccup or network glitch can introduce an error.

Here is my plan.

Now that I have asset dividend calculations precise at any block, I can build on that. Internally, each NXT acct with an asset tx history has every asset tx that it ever did in a linear list. The number of transactions for any given user will be small compared to CPU speed, so linear is fine and so will recalculating everything from scratch.

Instead of a complex state machine that needs to make sure it has grabbed different events at different places in the right order to prevent any double transfers but to always make sure exactly one transfer is made, I can scan the list of asset tx and see if anything should be done. I think this greatly simplifies things as I can even now visualize the possible variations.

Ignoring how many confirmations a tx has, we have these trigger conditions:
A) external blockchain deposit but no corresponding NXT asset xfer
B) NXT asset withdrawal, but no external blockchain payment

Really it seems so simple now, I wonder why I had problems with this? So, I am concerned that I am missing some cases.

Since each NXT asset tx already has an entry and I even already defined a *cointxid ptr (unused) in the structure, I guess I knew something like this was needed. I already have the list of external blockchain deposits and it has fields to indicate completion, I think the simplifying factor here is that I will end up with just one list per NXT acct.

Since one of the three MGW will actually issue the asset xfer, the best way to sync the state of all three is for them to update the list with the binding of external blockchain deposit to NXT asset list when they see the asset xfer. Just need to make sure that the server that does the asset xfer doesnt do more than one. That means RAM based state variable for this and waiting N blocks on startup before doing any transfers to satisfy an external blockchain deposit.

If the above is good, we now have a totally synchronized state of deposits (after the asset xfer appears on the blockchain), during the time it is pending one of the MGW servers will have a different state. There is potential here of an "attack" by someone depositing all the time to get the three servers in a different state. To avoid this deposit attack, I think all I have to do is, nothing. As long as I am just scanning the list of the NXT acct's asset tx and dont credit any account until N blocks have passed, only what has appeared on the blockchain will affect the MGW's crediting of each account. The suppression of sending more than one transfer shouldnt affect the balance calculation, so I think I can ignore the "deposit attack"

OK, now I just need a deterministic way to select the NXT account to process. It seems a bit brute force, but 20,000 is a small number to a CPU and no asset has anywhere near that number of asset holders. I have a list of all asset tx for each asset, so I just scan that list for all NXT accts and scan each NXT acct's list for any required withdrawal processing. I need a timestamp of the block in which the withdrawal request came in (via asset xfer), then I can just process them in order of NXT acct #, or withdrawal size, doesnt matter as all three MGW servers should have the same lists

Once the three MGW servers are in agreement on the order of withdrawals that need to be processed, that solves the last sticking point. All external blockchain tx will map onto a NXT asset tx. I can verify that no tx is processed until N confirmations. I think processing one withdraw at a time will be fast enough as it can do several per minute, which is thousands of withdraws per day.

If the above is good, I can code it up and finally release it for testing.
Feedback please. MGW is super important and I want to have more than just me to verify the highlevel algo

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

valarmg

  • Hero Member
  • *****
  • Karma: +178/-57
  • Offline Offline
  • Posts: 1766
    • View Profile
Re: Multigateway status reports
« Reply #153 on: May 23, 2014, 08:06:39 pm »

Let me see if I can understand this. You only have two MGW trigger events. Deposit + Withdrawal. Deposit=Deposit from external coin and asset is added to NXT account. Withdrawal=Asset is withdrawn from AE, and coin is sent to external wallet.

A. Deposit.
Deposit request occurs (external request sent to MWG).
Multisig wallet is created, and funds are lodged in that wallet.
When confirmed, coin deposit details are entered into central list, agreed upon by the 3MGW.
One of the MGWs does the asset deposit, using a state machine to make sure it's only done once.

B Withdrawal.
The MGW is continually polling all the NXT addresses, looking for withdrawals from MGW assets.
When found, they are added to a central list, once again agreed by all 3 MGWs, order is also agreed.
Going through the list one by one, funds are sent from central MGW wallet back onto external blockchain.
Logged
NXT-CSED-4PK5-AR4V-6UB5V

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #154 on: May 23, 2014, 08:21:26 pm »

Let me see if I can understand this. You only have two MGW trigger events. Deposit + Withdrawal. Deposit=Deposit from external coin and asset is added to NXT account. Withdrawal=Asset is withdrawn from AE, and coin is sent to external wallet.

A. Deposit.
Deposit request occurs (external request sent to MWG).
Multisig wallet is created, and funds are lodged in that wallet.
When confirmed, coin deposit details are entered into central list, agreed upon by the 3MGW.
One of the MGWs does the asset deposit, using a state machine to make sure it's only done once.

B Withdrawal.
The MGW is continually polling all the NXT addresses, looking for withdrawals from MGW assets.
When found, they are added to a central list, once again agreed by all 3 MGWs, order is also agreed.
Going through the list one by one, funds are sent from central MGW wallet back onto external blockchain.
There is a third significant event, the request for deposit addresses. until someone gets a deposit address, MGW basically cant do anything for them. The user publishes AM that requests a multisig deposit addr, the MGW servers coordinate and one of them pubishes AM with the multisig deposit addr

Now, MGW is also monitoring all bitcoind blocks (and LTC, DOGE, DRK, etc) and when it sees a multisig address it has issued, this triggers a transfer of assets to the user's acct corresponding with the deposit

I just take the (NXT addr % NUM_GATEWAYS) to choose which gateway does the actual asset transfer, but all three are independently doing all the other calculations. They all have independently updated asset lists and NXT acct specific asset tx list

MGW does not need to monitor all NXT addr, just all NXT blocks and whenever there is a transfer of a BTC asset back to the MGW, this creates a withdrawal request. Hopefully the asset xfer that did this has a comment field validating destination, if not it will default to an address that was registered when requesting multisig deposit address

The reason why I dont think MGW will be a tempting target for hackers is that there is NO CENTRAL WALLET! Internally I create a virtual wallet that is composed of the sum of all unspent outputs from all the depositor's multisig accts. That also means no sweep transaction, as soon as a deposit gets enough confirmations, the MGW virtual balance is adjusted.

Here comes the part that automatically establishes MGW consensus. Not sure if you know about rawtransactions and how to make them. Trust me, it is not as easy as it sounds. Every detail of the tx needs to be specified in the precise syntax. Then at the end of it you get a bunch of bytes that is the transaction. It is these bytes that all MGW servers need to agree on, exactly. So that means every detail of the tx is the same. This means the order of withdrawal processing is critical.You need a deterministic algo to define all the tx parameters and it has to end up with the identical rawtx bytes on all three MGW server.

To make it even less of a target, over time, all the MGW multisig deposit addresses will converge to the average balance. Even if someone deposits 100 BTC or 0.1 BTC, over time they will both end up with around the average deposit. I am guessing around 1 BTC, maybe a bit more.

So a hacker would have to somehow compromise servers that are independently managed and then construct the rawtx bytes, get them signed to be able to steal 1 BTC. With active monitoring by the other servers and nodes running the MGW client, even if this unlikely event happens,we will find out right away and send in the dobermans

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

valarmg

  • Hero Member
  • *****
  • Karma: +178/-57
  • Offline Offline
  • Posts: 1766
    • View Profile
Re: Multigateway status reports
« Reply #155 on: May 24, 2014, 11:26:27 am »

I'm confused on one point. From what I'm reading, you seem to be saying that a withdrawal request is initiated from the AE. Is that right? However, it seems to me that from within the AE, it's only possible to transfer the asset around between NXT accounts, and that to withdraw, you'll have to initiate a redeem request via the MGW gui.

Edit: Ok, I think I understand. If (within AE) you transfer the asset to the MGW NXT account, then that triggers a withdrawal request. If so, two questions. 1) Can't you just monitor the MWG NXT account looking for transactions, rather than all the NXT blocks? and 2) What happens if a NXT account that has never deposited does the withdrawal (the asset was just transferred to them), they'll then have no deposit address?
« Last Edit: May 24, 2014, 11:41:32 am by valarmg »
Logged
NXT-CSED-4PK5-AR4V-6UB5V

chanc3r

  • Hero Member
  • *****
  • Karma: +124/-50
  • Offline Offline
  • Posts: 1019
  • NXTInspect
    • View Profile
Re: Multigateway status reports
« Reply #156 on: May 24, 2014, 11:41:47 am »

I'm confused on one point. From what I'm reading, you seem to be saying that a withdrawal request is initiated from the AE. Is that right? However, it seems to me that from within the AE, it's only possible to transfer the asset around between NXT accounts, and that to withdraw, you'll have to initiate a redeem request via the MGW gui.

You would initiate a withdraw by sending the asset back to the ASSET ISSUER NXT account owned by MGW - in order to balance the MGW would need to transfer the real coins from the multi-sig wallet to your registered withdrawal address.
Logged
NXT: 29996814460165 (NXT-JTA7-B2QR-8BFC-2V222)
@imrimr @NXTinspect

valarmg

  • Hero Member
  • *****
  • Karma: +178/-57
  • Offline Offline
  • Posts: 1766
    • View Profile
Re: Multigateway status reports
« Reply #157 on: May 24, 2014, 11:45:09 am »

I'm confused on one point. From what I'm reading, you seem to be saying that a withdrawal request is initiated from the AE. Is that right? However, it seems to me that from within the AE, it's only possible to transfer the asset around between NXT accounts, and that to withdraw, you'll have to initiate a redeem request via the MGW gui.

You would initiate a withdraw by sending the asset back to the ASSET ISSUER NXT account owned by MGW - in order to balance the MGW would need to transfer the real coins from the multi-sig wallet to your registered withdrawal address.

Thanks. (See my edit above, I have two further questions)
1) Can't you just monitor the MWG NXT account looking for transactions, rather than all the NXT blocks? and 2) What happens if a NXT account that has never deposited does the withdrawal (the asset was just transferred to them), they'll then have no deposit address?
Logged
NXT-CSED-4PK5-AR4V-6UB5V

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #158 on: May 24, 2014, 02:35:32 pm »

I'm confused on one point. From what I'm reading, you seem to be saying that a withdrawal request is initiated from the AE. Is that right? However, it seems to me that from within the AE, it's only possible to transfer the asset around between NXT accounts, and that to withdraw, you'll have to initiate a redeem request via the MGW gui.

You would initiate a withdraw by sending the asset back to the ASSET ISSUER NXT account owned by MGW - in order to balance the MGW would need to transfer the real coins from the multi-sig wallet to your registered withdrawal address.

Thanks. (See my edit above, I have two further questions)
1) Can't you just monitor the MWG NXT account looking for transactions, rather than all the NXT blocks? and 2) What happens if a NXT account that has never deposited does the withdrawal (the asset was just transferred to them), they'll then have no deposit address?
In scenarios where there are more than a certain number of accounts, it is much less work to just decode each block and all the tx inside it than poll every NXT acct to see if it got any changes.
Additionally NXTservices is tracking all assets, so much simpler and less API calls to just process all blocks

Withdrawals MUST be done from a preregistered NXT acct, the one that requested the multisig address
The withdraw function currently creates a tokenized request that can only be created by the NXT acct that is requesting the withdraw. Otherwise anybody can withdraw anybody elses funds. I thought that would be bad.

So that is why it is a two step process. You click the button to get the URL and paste it into the browser. Of course, we need a GUI to automate this, but private key always stays local.
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

lyynx

  • Jr. Member
  • **
  • Karma: +17/-0
  • Offline Offline
  • Posts: 83
    • View Profile
    • CoinWith.me
Re: Multigateway status reports
« Reply #159 on: May 26, 2014, 01:30:52 pm »

Keep up the great work, I think this will be one of the biggest additions to Nxt.
Logged
NxtDex.com - Directory of Nxt Related Assets, Shops and Services.

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #160 on: May 27, 2014, 09:10:26 pm »

NXTservices_v05 due out tonight. It will have rewritten MGW deposits and needs testers to create DOGE deposits
See https://nxtforum.org/nxtservices-releases/nxtservices_v05-added-mgw-deposits/msg30207/#msg30207 for more details.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

allwelder

  • Hero Member
  • *****
  • Karma: +196/-13
  • Offline Offline
  • Posts: 1867
  • NxtChina.org
    • View Profile
    • NxtChina.org
Re: Multigateway status reports
« Reply #161 on: May 29, 2014, 01:43:16 am »

Run NXTservices_v05.exe with win32 ,
displayed as following
Could not connect server,what's the problem?
Logged
NxtChina |Weibo |Twitter Donation welcomed:NXT-APL9-66GU-K8LY-B3JJJ

longzai1988

  • Sr. Member
  • ****
  • Karma: +44/-1
  • Offline Offline
  • Posts: 268
    • View Profile
Re: Multigateway status reports
« Reply #162 on: May 29, 2014, 01:52:52 am »

allwelder

You must also start Nxt client in testnet
Logged
10533725144527397164 NXT-HSBE-8PWL-CUCD-BHUD6
NXT Multigateway - http://multigateway.com/

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #163 on: May 29, 2014, 04:42:59 am »

Finally got things setup to test withdraws and manually verified the multisig still works.
So now its the home stretch, just need to get three MGW server consensus on the withdraw and then sign and broadcast and post AM. I actually already did all this, so just a matter of using the existing code in a new way.

It really feels like I am in the home stretch. With the new redesign, I am clear on exactly what needs to be done, so I hope that by Friday I will have a withdraw enabled MGW, which means it will be a beta release.

Then while I am gone, I hope people will really test it so I can fix any issues and make a mainnet release

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

allwelder

  • Hero Member
  • *****
  • Karma: +196/-13
  • Offline Offline
  • Posts: 1867
  • NxtChina.org
    • View Profile
    • NxtChina.org
Re: Multigateway status reports
« Reply #164 on: May 29, 2014, 07:48:57 am »

allwelder

You must also start Nxt client in testnet
forget the most important procedure. ;D
working now.
Thanks.
Logged
NxtChina |Weibo |Twitter Donation welcomed:NXT-APL9-66GU-K8LY-B3JJJ

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #165 on: May 29, 2014, 07:51:21 pm »

I need somebody to verify the min txfee for BTC, LTC, DOGE and DRK

Now, please verify that this min txfee applies to multisig tx also.

Thanks. I just have to change a #define, but if I put the wrong amount in, it seems the tx is rejected.

I am also setting the MGW fee to match the mintx fee, so it will cost double normal txfee to withdraw. Still less than almost all exchanges and I think this will give MGW enough revenue to pay for servers and maybe even dividends. Not sure though, so dont go crazy on AE. It could still just be breakeven economics

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

VanBreuk

  • Hero Member
  • *****
  • Karma: +362/-19
  • Offline Offline
  • Posts: 2772
    • View Profile
Re: Multigateway status reports
« Reply #166 on: May 29, 2014, 08:23:29 pm »

You mean the minimum usual tx fees so the tx will be included most certainly in the next block? Because at least in BTC there are recommended fees, but not really minimum fees set in stone. Factors are tx weight and priority.
Logged
GPG Fingerprint: B020 D1C1 F289 3B2C 3577  9EAD 455D D175 5913 C7F1

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #167 on: May 29, 2014, 08:29:40 pm »

You mean the minimum usual tx fees so the tx will be included most certainly in the next block? Because at least in BTC there are recommended fees, but not really minimum fees set in stone. Factors are tx weight and priority.
I think most people would prefer to pay a little extra to make sure their tx gets included as quick as possible. After all they decided to withdraw, so there must be some sort of urgency, otherwise they would have just left it in.

So, what txfee would give us 99% chance of being included in the next block
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

VanBreuk

  • Hero Member
  • *****
  • Karma: +362/-19
  • Offline Offline
  • Posts: 2772
    • View Profile
Re: Multigateway status reports
« Reply #168 on: May 29, 2014, 09:24:05 pm »

Default fee in BTC is 0.0001 and that's what you will find in most txs. Would it be possible for users to have the option of a normal fee of 0.0001, and a high priority fee of slightly more than the average to virtually ensure inclusion in the next block? I don't have average fee stats, and perhaps the difference is as little as 90% > 99%, but it would be accurate.

Otherwise, just set it at 0.0001 BTC for the default.
« Last Edit: May 29, 2014, 09:26:16 pm by VanBreuk »
Logged
GPG Fingerprint: B020 D1C1 F289 3B2C 3577  9EAD 455D D175 5913 C7F1

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #169 on: May 29, 2014, 09:32:41 pm »

Default fee in BTC is 0.0001 and that's what you will find in most txs. Would it be possible for users to have the option of a normal fee of 0.0001, and a high priority fee of slightly more than the average to virtually ensure inclusion in the next block? I don't have average fee stats, and perhaps the difference is as little as 90% > 99%, but it would be accurate.

Otherwise, just set it at 0.0001 BTC for the default.
I think XCP is having to pay more than that...
Also,need exact txfees for other coins
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

valarmg

  • Hero Member
  • *****
  • Karma: +178/-57
  • Offline Offline
  • Posts: 1766
    • View Profile
Re: Multigateway status reports
« Reply #170 on: May 29, 2014, 09:41:33 pm »

For Dogecoin, seems that minimum fee is one doge, and that fee worked with the MGW multisig address.
Logged
NXT-CSED-4PK5-AR4V-6UB5V

VanBreuk

  • Hero Member
  • *****
  • Karma: +362/-19
  • Offline Offline
  • Posts: 2772
    • View Profile
Re: Multigateway status reports
« Reply #171 on: May 29, 2014, 09:56:12 pm »

I think the formula nodes approve is something like

Minimum Fee = Basic Fee * [1 + (INT(tx bytes/1000))]

If transactions are lighter than than 1 kB, Minimum Fee = Basic Fee

And basic fees would be 0.0001 for BTC, 1 for DOGE, 0.01 for LTC, 0.001 for DRK, and currently (I had to check this) 0.025 DRK fee in the case of collateral transactions (darksend)

Minimum fees can be higher for tx with very small amounts to prevent blockchain dust.
Logged
GPG Fingerprint: B020 D1C1 F289 3B2C 3577  9EAD 455D D175 5913 C7F1

DoM P

  • Hero Member
  • *****
  • Karma: +114/-147
  • Offline Offline
  • Posts: 1115
    • View Profile
    • Crypto Finance Analysis Consulting
Re: Multigateway status reports
« Reply #172 on: June 05, 2014, 07:30:53 am »

Could one of you guys summarize what is MultiGateway for me please ?

I'd like to include it to the document I am creating for investors.

Thanks
Logged
Have you heard of CryptoCoins? You should!
Crypto Finance Analysis Consulting: cfa-consulting.ch
Nxt : NXT-LP8G-9NHV-VUQB-58ZZF

valarmg

  • Hero Member
  • *****
  • Karma: +178/-57
  • Offline Offline
  • Posts: 1766
    • View Profile
Re: Multigateway status reports
« Reply #173 on: June 05, 2014, 07:43:27 am »

Could one of you guys summarize what is MultiGateway for me please ?

I'd like to include it to the document I am creating for investors.

Thanks

It's a software bridge between other cryptocurrencies and the Nxt asset exchange. It allows BTC, LTC, DOGE, etc to be deposited onto the asset exchange and traded for NXT.

(Once the asset exchange allows asset-to-asset trading, then trading will be possible between these currencies and each other and other assets without going through NXT)

There's a graphic here:
https://nxtforum.org/nxtservices-releases/how-to-test-multigateway-with-nxtservices-a-test-user-guide/
Logged
NXT-CSED-4PK5-AR4V-6UB5V

DoM P

  • Hero Member
  • *****
  • Karma: +114/-147
  • Offline Offline
  • Posts: 1115
    • View Profile
    • Crypto Finance Analysis Consulting
Re: Multigateway status reports
« Reply #174 on: June 05, 2014, 07:46:24 am »

Could one of you guys summarize what is MultiGateway for me please ?

I'd like to include it to the document I am creating for investors.

Thanks

It's a software bridge between other cryptocurrencies and the Nxt asset exchange. It allows BTC, LTC, DOGE, etc to be deposited onto the asset exchange and traded for NXT.

(Once the asset exchange allows asset-to-asset trading, then trading will be possible between these currencies and each other and other assets without going through NXT)

There's a graphic here:
https://nxtforum.org/nxtservices-releases/how-to-test-multigateway-with-nxtservices-a-test-user-guide/

Thanks a lot. I had an unclear idea. Better now.
Logged
Have you heard of CryptoCoins? You should!
Crypto Finance Analysis Consulting: cfa-consulting.ch
Nxt : NXT-LP8G-9NHV-VUQB-58ZZF

jabo38

  • Sr. Member
  • ****
  • Karma: +40/-38
  • Offline Offline
  • Posts: 381
    • View Profile
Re: Multigateway status reports
« Reply #175 on: June 05, 2014, 02:34:56 pm »

Keep up the great work, I think this will be one of the biggest additions to Nxt.

THE BIGGEST since BCNext made his first post
Logged
Never Enough Money

mikesbmw

  • Hero Member
  • *****
  • Karma: +41/-6
  • Offline Offline
  • Posts: 590
    • View Profile
Re: Multigateway status reports
« Reply #176 on: June 08, 2014, 09:54:41 am »

Guys,

was just wondering about this, is all of MGW completely open source?
Do we know what software and infrastructure is used?

Is the software (and implementation) being reviewed?
Logged
Number one Nxt nerd with fancy fantasies!
NXT: NXT-5MMT-RUYC-NCC9-AU8BL

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #177 on: June 08, 2014, 01:41:07 pm »

Guys,

was just wondering about this, is all of MGW completely open source?
Do we know what software and infrastructure is used?

Is the software (and implementation) being reviewed?
Mgw is being reviewed
Almost all mgw is open source, but keeping a very small amount internal until it is at least in production to prevent premature cloning
Other than standard libraries, i have written all of mgw
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

allwelder

  • Hero Member
  • *****
  • Karma: +196/-13
  • Offline Offline
  • Posts: 1867
  • NxtChina.org
    • View Profile
    • NxtChina.org
Re: Multigateway status reports
« Reply #178 on: June 08, 2014, 02:19:49 pm »

Guys,

was just wondering about this, is all of MGW completely open source?
Do we know what software and infrastructure is used?

Is the software (and implementation) being reviewed?
Mgw is being reviewed
Almost all mgw is open source, but keeping a very small amount internal until it is at least in production to prevent premature cloning
Other than standard libraries, i have written all of mgw
thumb up
Logged
NxtChina |Weibo |Twitter Donation welcomed:NXT-APL9-66GU-K8LY-B3JJJ

mikesbmw

  • Hero Member
  • *****
  • Karma: +41/-6
  • Offline Offline
  • Posts: 590
    • View Profile
Re: Multigateway status reports
« Reply #179 on: June 08, 2014, 07:05:30 pm »

Guys,

was just wondering about this, is all of MGW completely open source?
Do we know what software and infrastructure is used?

Is the software (and implementation) being reviewed?
Mgw is being reviewed
Almost all mgw is open source, but keeping a very small amount internal until it is at least in production to prevent premature cloning
Other than standard libraries, i have written all of mgw
Cool!

Is it being reviewed by someone we know?
Logged
Number one Nxt nerd with fancy fantasies!
NXT: NXT-5MMT-RUYC-NCC9-AU8BL

Frohike

  • Full Member
  • ***
  • Karma: +30/-1
  • Offline Offline
  • Posts: 121
  • Help to build NXT and make dreams come true
    • View Profile
    • http://multigateway.org
Re: Multigateway status reports
« Reply #180 on: June 08, 2014, 08:42:00 pm »

Hi,

We are working on the infrastructure part.
I can tell you it is going to be a very cool setup, but I can't elaborate on this. Everything I say, is information which could be useful for someone with bad intentions.
We're securing it very, very well, and one of the policies I'd say is not explaining what we're using and how we do it. That would only help black hats when trying to get in.

I'm working on it everyday, and it's only like 10% building the environment and 90% working on security.
Logged
Security, Linux, Hardening, Infrastructure services. *** Currently working for NXTservices, like the MGW and Instantdex. ***---Also working on SuperNET - SuperNET server and services---***

NXT-TJ66-UWW7-GKC9-3F3PE

mikesbmw

  • Hero Member
  • *****
  • Karma: +41/-6
  • Offline Offline
  • Posts: 590
    • View Profile
Multigateway status reports
« Reply #181 on: June 08, 2014, 09:53:41 pm »

Hee! Where's the answer from James and my question gone?

Edit: Nevermind it's Tapatalk screwing things up...
Thanks for your answer!
« Last Edit: June 08, 2014, 09:57:58 pm by mikesbmw »
Logged
Number one Nxt nerd with fancy fantasies!
NXT: NXT-5MMT-RUYC-NCC9-AU8BL

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #182 on: June 09, 2014, 11:37:17 pm »

I found a memory leak which could have caused the memory leaking until it consumed all memory for the version I rushed out the door before my week off.

Still waiting for the bitcoind to reindex, but hopefully the production server guys can verify it is stable.
I also put in an improvement that boosts the number of withdrawals up significantly from 100/day
at least I think it should. Even 100 withdraws per day would be sufficient until we hit pretty high volumes, but I know I hate to wait anymore than I have to.

Not sure of the throughput, but I think it is possible to get 1000+ withdraws per day
I think i fixed the dividend calc bug (seems to be a MS compiler bug!) so if the MGW server is stable, I can make a new v07 NXTservices that uses the production servers. Still testnet, but at least the servers wont die if my local ISP drops my internet connection and I lose ssh (happens daily it seems)

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #183 on: June 10, 2014, 06:00:32 am »

It seems the memory leak is fixed, at least it looks very promising. A production server is running MGW and it has been stable for hours

I am testing a withdraw using production parameters and do you know how long 3 BTC blocks takes? FOREVER!
After getting used to 1 minute blocks, waiting for 3 BTC blocks is just so slow...

Anyway, this avoids all sorts of malleability issues and reduces the possible error handling cases dramatically. So, it will be a half hour wait for BTC deposit/withdrawal to register.

Now if I got it to update the withdraw details via AM instead of waiting for it appearing on the BTC blockchain, then we can do 1 withdraw per NXT block, or 1 every 2 at worst. Much better than relying on BTC blockchain to sync the MGW server states.

Now I could sync things via direct connect, but 500 to 1000 withdraws per day seems plenty and it is much more realiable this way.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

Frohike

  • Full Member
  • ***
  • Karma: +30/-1
  • Offline Offline
  • Posts: 121
  • Help to build NXT and make dreams come true
    • View Profile
    • http://multigateway.org
Re: Multigateway status reports
« Reply #184 on: June 10, 2014, 09:00:19 am »

Very nice James! Get us the new version and we can run it over there and see how it behaves.
I would like to monitor it, so I can precisely see how the system resources are used. I'm talking CPU, Mem, Disk activity and network activity.
We might need to optimize things before going into production. Also, we do want proactive capacity planning, we need to know our limits.

In the mean time I'm very busy with a parallel project for our NXT MGW setup. When I'm done, this configuration management tool will be a breeze for us, setting up new servers and keep all servers in sync. This configuration management infrastructure will help us bring new servers online in a matter of minutes. Also, from an administrative point of view, the configuration and management can be done from a central point.

If other NXT ventures / ideas will rise up which need infra, let me know. I can create a new configuration pool for those servers and manage them the same way.
It's a nifty way to have scalability, security and administrative ease in one fell swoop. This fits right in with NXT, and this way we can keep up with the fast growth of ideas and react abruptly when more and more enthousiasts are entering the NXT arena.

Logged
Security, Linux, Hardening, Infrastructure services. *** Currently working for NXTservices, like the MGW and Instantdex. ***---Also working on SuperNET - SuperNET server and services---***

NXT-TJ66-UWW7-GKC9-3F3PE

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #185 on: June 10, 2014, 09:07:55 am »

Very nice James! Get us the new version and we can run it over there and see how it behaves.
I would like to monitor it, so I can precisely see how the system resources are used. I'm talking CPU, Mem, Disk activity and network activity.
We might need to optimize things before going into production. Also, we do want proactive capacity planning, we need to know our limits.

In the mean time I'm very busy with a parallel project for our NXT MGW setup. When I'm done, this configuration management tool will be a breeze for us, setting up new servers and keep all servers in sync. This configuration management infrastructure will help us bring new servers online in a matter of minutes. Also, from an administrative point of view, the configuration and management can be done from a central point.

If other NXT ventures / ideas will rise up which need infra, let me know. I can create a new configuration pool for those servers and manage them the same way.
It's a nifty way to have scalability, security and administrative ease in one fell swoop. This fits right in with NXT, and this way we can keep up with the fast growth of ideas and react abruptly when more and more enthousiasts are entering the NXT arena.
PM me the IP addresses and NXT accts to use for the production servers. I have different sets of numbers...
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #186 on: June 10, 2014, 09:14:31 am »

Very nice James! Get us the new version and we can run it over there and see how it behaves.
I would like to monitor it, so I can precisely see how the system resources are used. I'm talking CPU, Mem, Disk activity and network activity.
We might need to optimize things before going into production. Also, we do want proactive capacity planning, we need to know our limits.

In the mean time I'm very busy with a parallel project for our NXT MGW setup. When I'm done, this configuration management tool will be a breeze for us, setting up new servers and keep all servers in sync. This configuration management infrastructure will help us bring new servers online in a matter of minutes. Also, from an administrative point of view, the configuration and management can be done from a central point.

If other NXT ventures / ideas will rise up which need infra, let me know. I can create a new configuration pool for those servers and manage them the same way.
It's a nifty way to have scalability, security and administrative ease in one fell swoop. This fits right in with NXT, and this way we can keep up with the fast growth of ideas and react abruptly when more and more enthousiasts are entering the NXT arena.
NXTsync will need a set of servers, also a couple of pool servers for the NXTcoinsco coins and the most would be the NXTmixer servers that will need to be renamed to NXTprivacy servers. I will get you more details as I figure them out
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #187 on: June 13, 2014, 04:08:06 pm »

MGW is in beta test and I have fixed all the reported bugs. Even the overlapped withdraw test worked
some possible wonkiness happened when some of the servers were out of sync due to out of date acct numbers, so maybe there is a bug or two left.

Next step is to get the production servers online. Separately, I have made "production" MGW version that still uses testnet, but uses the production servers (at least the test production servers)
chanc3r has the latest source code and is reviewing it

So what is left to do is:
1) beta test, find bugs, fix bugs (?)
2) get test production servers running MGW (<1 day)
3) make mainnet MGW (probably good to make it able to switch between testnet and mainnet) (<1 day)
4) make improvements from source review (?)
5) get final production servers running MGW (<1 day)

The times in () are how much time it will take me to do and not the time for others. Regardless, the finish line is in sight!

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #188 on: June 16, 2014, 10:15:24 am »

Guys,

was just wondering about this, is all of MGW completely open source?
Do we know what software and infrastructure is used?

Is the software (and implementation) being reviewed?
Mgw is being reviewed
Almost all mgw is open source, but keeping a very small amount internal until it is at least in production to prevent premature cloning
Other than standard libraries, i have written all of mgw
Cool!

Is it being reviewed by someone we know?
chanc3r and hopefully ChuckOne will get time to do it too
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

Steve

  • Sr. Member
  • ****
  • Karma: +28/-0
  • Offline Offline
  • Posts: 349
  • Steve please
    • View Profile
Re: Multigateway status reports
« Reply #189 on: June 16, 2014, 06:10:13 pm »

This is amazing, you guys are genius! How the heck you do all this I dont know.

Will this be open for public use? NXT gateways would allow for nonbiased currency (cant remember the term right now..) payments!?

<fake-potential-scenario>

Hey want to accept bitcoins at your business? Great! We have a way for you to accept just about anything.

</fake-potential-scenario>
« Last Edit: June 16, 2014, 06:19:24 pm by Steve »
Logged

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #190 on: June 17, 2014, 03:02:59 am »

This is amazing, you guys are genius! How the heck you do all this I dont know.

Will this be open for public use? NXT gateways would allow for nonbiased currency (cant remember the term right now..) payments!?

<fake-potential-scenario>

Hey want to accept bitcoins at your business? Great! We have a way for you to accept just about anything.

</fake-potential-scenario>
Using NXT AE + MGW, any business can accept "BTC" simply by accepting the mgwBTC. All any business would need is a NXT acct. Then customers can simply transfer assets to the business with the appropriate comment filled out and just by clicking on the web GUI can manually process all the orders.

Of course, it wont be so hard to automate the processing of the asset transfers (I can probably add some API calls to simplify it even further) but it seems like it is much easier to convince a business to start accepting crypto if all they had to do was download the standard NXT web GUI.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

anon136

  • Hero Member
  • *****
  • Karma: +86/-19
  • Offline Offline
  • Posts: 1015
    • View Profile
Re: Multigateway status reports
« Reply #191 on: June 18, 2014, 02:36:19 am »

can you use AT to make multigateway completely trustless? (once at is finished of course)
Logged

Daedelus

  • Hero Member
  • *****
  • Karma: +230/-12
  • Offline Offline
  • Posts: 3280
    • View Profile
Re: Multigateway status reports
« Reply #192 on: June 19, 2014, 01:15:28 pm »

FYI,
NAS Multigateway Plan


Source: https://bitcointalk.org/index.php?topic=523187.msg7396210#msg7396210

Not sure it actually explains much.
Logged
NXT: NXT-4CS7-S4N5-PTH5-A8R2Q

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #193 on: June 19, 2014, 05:58:53 pm »

can you use AT to make multigateway completely trustless? (once at is finished of course)
I dont see how it is possible
The fundamental issue is where do the bitcoin wallets and passwords go
If you can solve that, maybe there is a way to combine atomix tx and do it, but seems to be a big problem
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

xibeijan

  • Guest
Re: Multigateway status reports
« Reply #194 on: July 04, 2014, 06:59:42 pm »

Any development/testing news?
Logged

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #195 on: July 04, 2014, 07:28:18 pm »

Any development/testing news?
https://nxtforum.org/nxtservices-releases/how-to-test-multigateway-with-nxtservices-a-test-user-guide/1140/

just waiting for production servers, MGW has been running on autopilot for several days now
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

No_2

  • Newbie
  • *
  • Karma: +1/-0
  • Offline Offline
  • Posts: 10
  • BTC: the beginning of stake-based public resources
    • View Profile
    • MetaLair.org
Re: Multigateway status reports
« Reply #196 on: September 05, 2014, 08:39:36 am »

For a given 2-of-3 transaction which locks a users coins to the NXT Multigateway exchange, can these transactions be spent by 2-of-3 to any address or only back to the user's address?
Logged
Please Like or Follow MetaLair.
Twitter: @MetaLair | Facebook | Google+

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #197 on: September 05, 2014, 08:43:47 am »

For a given 2-of-3 transaction which locks a users coins to the NXT Multigateway exchange, can these transactions be spent by 2-of-3 to any address or only back to the user's address?
only to threeway validated withdraw addresses that specify the destination acct and have the funds available

the utxo in your deposit address are not bound to you specifically, all the utxos form a virtual wallet and all the assets are matching the utxos, so as long as each withdraw is matching the assets returned, then everything stays in balance
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

No_2

  • Newbie
  • *
  • Karma: +1/-0
  • Offline Offline
  • Posts: 10
  • BTC: the beginning of stake-based public resources
    • View Profile
    • MetaLair.org
Re: Multigateway status reports
« Reply #198 on: September 05, 2014, 09:23:28 am »

For a given 2-of-3 transaction which locks a users coins to the NXT Multigateway exchange, can these transactions be spent by 2-of-3 to any address or only back to the user's address?
only to threeway validated withdraw addresses that specify the destination acct and have the funds available

the utxo in your deposit address are not bound to you specifically, all the utxos form a virtual wallet and all the assets are matching the utxos, so as long as each withdraw is matching the assets returned, then everything stays in balance

Right, so I'm trying to understand if the three nodes overseeing the 2-of-3 can spend funds anywhere or if the transfer is locked to specific addresses. I'm guessing if 2 of the 3 nodes agree then they can spend the funds to any new address. This would be required because:

  • One of the three nodes disappeared and a new node needs to be found to oversee the locked in fund (ala starfish arms) so funds will need to be transferred to a new 2-of-3 transaction and address.
  • Anyone holding the right 'coloured' NXT can withdraw that to their chosen altcoin, so the address will be different.
« Last Edit: September 05, 2014, 01:50:42 pm by No_2 »
Logged
Please Like or Follow MetaLair.
Twitter: @MetaLair | Facebook | Google+

nextcafe

  • Full Member
  • ***
  • Karma: +10/-1
  • Offline Offline
  • Posts: 196
  • 2.0
    • View Profile
Re: Multigateway status reports
« Reply #199 on: September 13, 2014, 06:39:17 pm »

When is Litecoin back up? Can we still send transactions?
Logged
NXT-HL5A-UXNE-UE5H-BGHUA

Frohike

  • Full Member
  • ***
  • Karma: +30/-1
  • Offline Offline
  • Posts: 121
  • Help to build NXT and make dreams come true
    • View Profile
    • http://multigateway.org
Re: Multigateway status reports
« Reply #200 on: September 13, 2014, 06:51:03 pm »

So, that wiill be hopefully the coming week, in case problems arise with MGW maybe the week after that.

I'm busy setting up MGW with BitHaus and MyNXT.info, hopefully finalizing most of the config this weekend and then next week configuration and testing.

MGW really needs to get up ASAP on the producation servers now and will handle the withdraw issue with LTC. 
Logged
Security, Linux, Hardening, Infrastructure services. *** Currently working for NXTservices, like the MGW and Instantdex. ***---Also working on SuperNET - SuperNET server and services---***

NXT-TJ66-UWW7-GKC9-3F3PE

allwelder

  • Hero Member
  • *****
  • Karma: +196/-13
  • Offline Offline
  • Posts: 1867
  • NxtChina.org
    • View Profile
    • NxtChina.org
Re: Multigateway status reports
« Reply #201 on: September 15, 2014, 10:21:43 am »

mgwBTC withdraw is also automatic or manual hand?
Logged
NxtChina |Weibo |Twitter Donation welcomed:NXT-APL9-66GU-K8LY-B3JJJ

VanBreuk

  • Hero Member
  • *****
  • Karma: +362/-19
  • Offline Offline
  • Posts: 2772
    • View Profile
Re: Multigateway status reports
« Reply #202 on: September 15, 2014, 10:40:15 am »

mgwBTC withdraw is also automatic or manual hand?

Withdraws in MGW are automated. Only in cases where there's a problem server operators may have to intervene to push a transaction forward.
Logged
GPG Fingerprint: B020 D1C1 F289 3B2C 3577  9EAD 455D D175 5913 C7F1

huafei

  • Jr. Member
  • **
  • Karma: +1/-1
  • Offline Offline
  • Posts: 66
    • View Profile
Re: Multigateway status reports
« Reply #203 on: November 02, 2014, 09:34:35 am »

BTC deposit and withdraw take hours in process at the MGW service between BTC blockchain and nxt blockchain.
Logged

Ikonic

  • Full Member
  • ***
  • Karma: +35/-6
  • Offline Offline
  • Posts: 224
    • View Profile
Re: Multigateway status reports
« Reply #204 on: November 13, 2014, 02:32:20 am »

I have been reluctant to use MGW for the past month or so. Is there an update on status? This is a great service that I would love to be able to use and support.


Ike
Logged
SAAS Asset Id: 14225517118742712534

TwinWinNerD

  • Hero Member
  • *****
  • Karma: +222/-116
  • Offline Offline
  • Posts: 2012
  • CEO BitPanda.com
    • View Profile
Re: Multigateway status reports
« Reply #205 on: November 13, 2014, 02:38:49 am »

I have been reluctant to use MGW for the past month or so. Is there an update on status? This is a great service that I would love to be able to use and support.


Ike

not only an update, but an amazing update!

https://nxtforum.org/lite-multigateway-releases/supernet-v1-3-2-2/

MGW now runs on productions servers, that means that they will be much more reliable and fast! Check it out

Ikonic

  • Full Member
  • ***
  • Karma: +35/-6
  • Offline Offline
  • Posts: 224
    • View Profile
Re: Multigateway status reports
« Reply #206 on: November 13, 2014, 03:46:57 am »

I have been reluctant to use MGW for the past month or so. Is there an update on status? This is a great service that I would love to be able to use and support.


Ike

not only an update, but an amazing update!

https://nxtforum.org/lite-multigateway-releases/supernet-v1-3-2-2/

MGW now runs on productions servers, that means that they will be much more reliable and fast! Check it out


Awesome news.


Thanks
Ike
Logged
SAAS Asset Id: 14225517118742712534

yns1971

  • Jr. Member
  • **
  • Karma: +1/-0
  • Offline Offline
  • Posts: 16
    • View Profile
Re: Multigateway status reports
« Reply #207 on: January 12, 2015, 06:55:40 am »

I wanted to try out the multigateway so withdrew to a bitcoin address in transaction 5254992471851427627. 1 day later, still no bitcoin. Is there a delay?

Transaction 5254992471851427627

{"redeem":"BTC","withdrawaddr":"1Ergw2TtJHgDEMNpxQUgNwFQCbLrPzG4EF","InstantDEX":""}
Logged

creemej

  • Jr. Member
  • **
  • Karma: +1/-12
  • Offline Offline
  • Posts: 22
    • View Profile
Re: Multigateway status reports
« Reply #208 on: April 28, 2015, 07:32:48 am »

Same for me.  Is there a delay?

Transaction 14773844296183657784

{"redeem":"BTC","withdrawaddr":"34KihLaDpNKCsFYnVcfpixsqH461iHqdQK","InstantDEX":""}

Nothing comes back, there is a man in the middle!!!

It must be a scam!!!
And if so, SuperNET and NXT are also part of it.

Who can proof me the contrary?

Logged

Frohike

  • Full Member
  • ***
  • Karma: +30/-1
  • Offline Offline
  • Posts: 121
  • Help to build NXT and make dreams come true
    • View Profile
    • http://multigateway.org
Re: Multigateway status reports
« Reply #209 on: April 28, 2015, 07:45:36 am »

it is delayed often,  but the proof that it is no scam is easy to see: no one ever did see his tx not going through.

Right now operator of node1 is working on his machine. That node is to be restarted probably. I'm in contact with him now.
Logged
Security, Linux, Hardening, Infrastructure services. *** Currently working for NXTservices, like the MGW and Instantdex. ***---Also working on SuperNET - SuperNET server and services---***

NXT-TJ66-UWW7-GKC9-3F3PE

Bernard Lerring

  • Sr. Member
  • ****
  • Karma: +30/-3
  • Offline Offline
  • Posts: 296
    • View Profile
Re: Multigateway status reports
« Reply #210 on: April 28, 2015, 10:21:28 am »

I'm also waiting for the last server to authorise. I have been informed that it should happen today sometime.

I'll update when it happens.

Edit: Still nothing on 28 Apr @ 21:57 UTC.

Edit: Transaction signed 29 Apr @~01:30 UTC.

It'll be good when the proposed server update is done and you aren't waiting days for signing. :)
« Last Edit: April 29, 2015, 07:51:24 am by Bernard Lerring »
Logged
Pages: 1 2 3 ... 11 [All]
 

elective-stereophonic
elective-stereophonic
assembly
assembly