elective-stereophonic
elective-stereophonic
Multigateway status reports
singapore
Please login or register.

Login with username, password and session length
Advanced search  

News:

Latest Stable Nxt Client: Nxt 1.12.2

Pages: 1 2 3 [4] 5 6 ... 11  All

Author Topic: Multigateway status reports  (Read 39667 times)

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #60 on: April 06, 2014, 12:13:49 am »

Don't get distracted. This 'product' is very important and huge!
It doesnt do me or anyone else any good, if I am not making good progress
there is a time for debugging
there is a time for coding
there is a time for designing
there is a time for brainstorming
there is no time for sleep

James

P.S. The secret project enhances multigateway, dont worry

I AM DYING TO KNOW WHAT IT IS!!!! :(
I hope to be able to announce significant details within a week

Today's tidbit is that Adam Levine, yes of Lets Talk bitcoin, has joined the team based on the idea and is also eagerly awaiting more details.

I have a couple more team members to assemble, alpha release to make to Wesley, then I will announce more details.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

AdamBLevine

  • Jr. Member
  • **
  • Karma: +5/-0
  • Offline Offline
  • Posts: 19
    • View Profile
Re: Multigateway status reports
« Reply #61 on: April 06, 2014, 12:21:41 am »

Happy to join the team (As an advisor until I understand how everything works and am convinced it's as brilliant as jl777 thinks it is.)
Logged

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #62 on: April 06, 2014, 02:10:58 am »

I've decided to put the multigateway interface into a websockets server. This will allow me to display account status and other info via API that is human readable and also easy for programs to access. I think this is why I had a hard time putting in all the instrumentation code for the multigateway. There was no easy way to actually look at the data.

This is why it is important not to force yourself, if something is holding you back, there usually is a reason. so now making requests to multigateway can be done via URL!

http://127.0.0.1:7777/multigateway?{"requestType":"register","DOGE":"<withdraw addr>","BTC":"<withdraw addr>"}

http://127.0.0.1:7777/multigateway?{"requestType":"withdraw","DOGE":"<any DOGE address>","amount":"12.456","TOTP":"123456"}

Now that I have a way to get commands in and out, I have a reference to code to and I also made it so that all the different NXTprotocol handlers I am writing plugs right into the websockets. Just change the "multigateway" to "<secret project>"

Now the one issue is that I am allergic to Windows. If anybody can help with getting the libwebsockets test-server compiled and running on Windows, that would be very helpful. Otherwise only Mac (and linux) users will be able to do the NXTprotocol API commands. I have made a few small changes to test-server.c, so as soon as someone can get a compile environment for libwebsockets http://libwebsockets.org/trac/libwebsockets, we can have a cross platform release

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #63 on: April 06, 2014, 03:07:38 am »

OK, I now have JSON commands from the browser. Surprising number of ways to mess up a simple pass through with so many layers.

Now I can actually make the JSON commands return useful info.

If anybody knows of a portable way to dynamically load and unload executable modules, let me know. It would be really nice to be able to swap in and out NXTprotocol handlers without having to recompile new configs

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #64 on: April 06, 2014, 03:35:54 am »

Do we need google authenticator? I think if I made it so that every request needs to have a NXT token generated, then it would prove the account holder (or at least someone with the password) is making the request. If someone else has the password, there is nothing multigateway can do anyway.

So, I propose making all important requests to be as follows:

http://127.0.0.1:7777/multigateway?[{"requestType":"important",...,"NXT":"1234567890123456789"},{"token":"<160 chars>"}], that way there is no ambiguity on what to tokenize. white space needs to be stripped. Just submit to NXT website token generator:

{"requestType":"important",...,"lastparm":"xyz"} as the website

I think this should be as secure as practical, essentially requiring account passphrase for all NXTprotocol API calls that are not just informational.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #65 on: April 06, 2014, 06:05:12 am »

Made a stripped down multigateway that can run locally. Seemed the most reliable way of verifying that the main multigateway servers are doing fine, plus I actually dont have a desktop version of Ubuntu, so only access to libwebsockets is on Mac.

Crammed two NXThandlers on top of NXTprotocol and both finally are reporting properly.

Now I can finally properly code the multigateway instrumentation and I can monitor it in realtime locally.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #66 on: April 06, 2014, 08:05:52 am »

How many people are running on Mac and available for testing.
Just putting in JSON URLs
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #67 on: April 06, 2014, 10:57:25 am »

400 lines of code later, multigateway JSON dump code is ready for debug. I just dump the entire internal state. once that is debugged, then it is a matter of making sure it stays self-consistent.

If you want to help, it would help if there was someone more than just me making sure all the JSON entries make sense. Its not so scary:

Code: [Select]
{
"gatewayid": -1,
"NXTaddr": "18232225178877143084",
"ipaddr": "181.47.159.125",
"NXTaccts": [{
"NXTaddr": "8737506522730586553",
"numassets": 1,
"assets": [{
"assetid": "15066647755760867565",
"quantity": "999.00000000",
"txids": [{
"txid": "14959687918642441123",
"sender": "16159101027034403504",
"receiver": "8737506522730586553",
"quantity": "999.00000000"
}]
}],
"numcoinaccts": 0
}, {
"NXTaddr": "2865886802744497404",
"numassets": 1,
"assets": [{
"assetid": "15066647755760867565",
"quantity": "998.00000000",
"txids": [{
"txid": "14682240803350035277",
"sender": "16159101027034403504",
"receiver": "2865886802744497404",
"quantity": "998.00000000"
}]
}],
"numcoinaccts": 0
}, {
"NXTaddr": "7126304194855053556",
"numassets": 0,
"numcoinaccts": 0
}, {
"NXTaddr": "16159101027034403504",
"numassets": 1,
"assets": [{
"assetid": "15066647755760867565",
"quantity": "-1997.00000000",
"txids": [{
"txid": "14959687918642441123",
"sender": "16159101027034403504",
"receiver": "8737506522730586553",
"quantity": "999.00000000"
}, {
"txid": "14682240803350035277",
"sender": "16159101027034403504",
"receiver": "2865886802744497404",
"quantity": "998.00000000"
}]
}],
"numcoinaccts": 0
}, {
"NXTaddr": "18232225178877143084",
"numassets": 0,
"numcoinaccts": 0
}, {
"NXTaddr": "1739068987193023818",
"numassets": 0,
"numcoinaccts": 0
}],
"coins": {
"DOGE": {
"assetname": "DOGE",
"funds": {
"deposits": "0.00000000",
"transfers": "0.00000000",
"withdraws": "0.00000000",
"moneysends": "0.00000000",
"remaining_transfers": "0.00000000",
"remaining_moneysends": "0.00000000",
"balance": "0.00000000"
},
"unspent": {
"maxunspent": "0.00000000",
"unspent": "0.00000000",
"maxavail": "0.00000000",
"minavail": "0.00000000",
"smallest_msigunspent": "0.00000000",
"confirmed": 0
},
"assetid": "13606501668992980954",
"serverport": "node2.mynxtcoin.org:22555",
"userpass": "dogecoinrpc:HSDU....",
"gateways": ["209.126.71.170", "209.126.73.156", "209.126.73.158"],
"coinid": 3,
"min_confirms": 3,
"pollseconds": 60,
"nummsigs": 0,
"numwithdraws": 0,
"numdeposits": 0,
"initdone": 0,
"blockheight": 170542,
"NXTheight": 0,
"txfee": "1.00010000",
"markeramount": "0.00010000"
},
"genaddr_txids": {
"numitems": 0,
"structsize": 224,
"hashsize": 100,
"numsearches": 0,
"numiterations": 0
},
"coin_txids": {
"numitems": 1,
"structsize": 112,
"hashsize": 100,
"numsearches": 2,
"numiterations": 2,
"numiterations": 1
},
"redeemtxids": {
"numitems": 0,
"structsize": 224,
"hashsize": 100,
"numsearches": 0,
"numiterations": 0
},
"gen depositaddr requests": {
"queuesize": 0
},
"call_gen_depositaddr": {
"queuesize": 0
},
"retryassetxfer": {
"queuesize": 0
},
"wait_for_NXTconfirms": {
"queuesize": 0
},
"process_withdraws": {
"queuesize": 0
},
"google authenticator": {
"queuesize": 0
},
"send coin": {
"queuesize": 0
}
}
}

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #68 on: April 06, 2014, 12:36:41 pm »

Just as I was waiting for test results we got assetoshis!
gotta do the whole beg for textNXT and reinitialize all the assets

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #69 on: April 06, 2014, 03:52:58 pm »

JSON outputs look decent, but cant test all the cases until testnet gets going again.
I did manage to get a generate deposits address (and withdraw address) with "website" key validation via websockets API, a two step process to generate the magic URL.

start with:
{%22requestType%22:%22genDepositaddrs%22,%22NXT%22:%2211445347041779652448%22,%22coins%22:{%22DOGE%22:%22DNyEii2mnvVL1BM1kTZkmKDcf8SFyqAX4Z%22}}

use the above to generate:
{%22token%22:%22bd855r204lqfskfg4que97uef8sh2blnh7ol9mcq2mmee1i2lu3a2ftnpqudlto04kqa15gpc14hu6c1dvs1p0vfjbkha5q423lqo8fbf5k0ketomirtkvveefqib5r03bl2pk2i3ju48dl7k4bpcqfehppch81h%22}

submit URL by combining:
http://127.0.0.1:7777/multigateway?[{%22requestType%22:%22genDepositaddrs%22,%22NXT%22:%2211445347041779652448%22,%22coins%22:{%22DOGE%22:%22DNyEii2mnvVL1BM1kTZkmKDcf8SFyqAX4Z%22}},{%22token%22:%22bd855r204lqfskfg4que97uef8sh2blnh7ol9mcq2mmee1i2lu3a2ftnpqudlto04kqa15gpc14hu6c1dvs1p0vfjbkha5q423lqo8fbf5k0ketomirtkvveefqib5r03bl2pk2i3ju48dl7k4bpcqfehppch81h%22}]

It looks much easier to read without %22 and with ", but somehow the browser is automatically converting.

anyway, this means that once I can get the libwebsockets compiled for Windows, there is no need for compiling a program as I will put in all the required controls in the API. Actually, other than being able to generate deposit addresses and set withdraw addresses is pretty much all the GUI the multigateway needs.

Next up is converting to assetoshis, and maybe I can finally get the withdraw side tested.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

kodtycoon

  • Hero Member
  • *****
  • Karma: +43/-18
  • Offline Offline
  • Posts: 916
    • View Profile
Re: Multigateway status reports
« Reply #70 on: April 06, 2014, 06:10:40 pm »

Just as I was waiting for test results we got assetoshis!
gotta do the whole beg for textNXT and reinitialize all the assets

James

james why not just ask who ever is restarting the testnet to send testnxt to an account and then pm you the passphrase.. just let it be the first thing they do after a restart so you dont loose time waiting for testnxt? or if you always have the same acc just get them to send it to you straight away as a priority after a restart? seems a silly thing to be loosing time over....
Logged

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #71 on: April 07, 2014, 01:49:47 am »

Just as I was waiting for test results we got assetoshis!
gotta do the whole beg for textNXT and reinitialize all the assets

James

james why not just ask who ever is restarting the testnet to send testnxt to an account and then pm you the passphrase.. just let it be the first thing they do after a restart so you dont loose time waiting for testnxt? or if you always have the same acc just get them to send it to you straight away as a priority after a restart? seems a silly thing to be loosing time over....
Forrest,
no worries, i do usually get testNXT pretty quickly
the bigger issue is that it seems to take about a day before testnet isnt wonky after a reset

However, this is why I like to have more than one project. I can always switch to a different project. Over the years I have found that it is impossible for everything a project needs to always be there exactly when you need it. It is like the difference between DOS (the microsoft OS remember that?) that could only do one task at a time. Sure it was 1% faster than a multitasking OS because it didnt have any overhead for switching processes, but start a disk I/O, wait forever,etc.

Unix type of multithreading always keeps the CPU busy.

Similarily, whenever the project I am working on has to wait for something (even myself if I cant figure out how to solve something), I switch. Some of the multigateway tests takes 10 minutes+ to run. If I just stared at the screen, i wouldnt be very productive, so I switch.

net result is that I am at least twice as productive as otherwise, but actually even more as the different projects enhance each other, so maybe it is more like a threefold boost. Then there is the hours I work, which is doubletime and I dont spend any time with meetings, commuting, etc, so that is another 2 to 3 times boost.

So, I am a simple C programmer with a 3*3 boost :)

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #72 on: April 07, 2014, 04:45:29 am »

Conversion to assetoshis is really simplifying things internally.
the need to be able to run a matching copy on user nodes that wont have write access to coin daemons is requiring me to convert a lot of internal functions to use coinid (just a number) instead of *cp (a pointer to coin daemon data structure). Since all the key transactions will appear in AM's, any NXT node will be able to keep track of any multigateway state changes.

Removing the assumption of having the coin daemon data is requiring going over all the code, but I had to do that due to assetoshis anyway.

Another long day in progress.

James

P.S. This Wed/Thu I will be offline due to real world stuff.
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

chanc3r

  • Hero Member
  • *****
  • Karma: +124/-50
  • Offline Offline
  • Posts: 1019
  • NXTInspect
    • View Profile
Re: Multigateway status reports
« Reply #73 on: April 07, 2014, 08:26:46 am »

I've tried to track James posts on this from BCT and here and summarise what the multi-sig gateway looks like so that I can understand it so comments on whether you think i've got it right would be most welcome because I don't want to waste James time to explain his genius to mere mortals like me.

This is my first attempt at it, I will continue this to further my own understanding but if useful to others I will continue to share.

https://dl.dropboxusercontent.com/u/53047281/multi-sig.doc

BR
Ian

PS I've also got a doc in progress on NodeCoin and NXT mixer but those are harder to decipher from James blogs ;)
Logged
NXT: 29996814460165 (NXT-JTA7-B2QR-8BFC-2V222)
@imrimr @NXTinspect

kodtycoon

  • Hero Member
  • *****
  • Karma: +43/-18
  • Offline Offline
  • Posts: 916
    • View Profile
Re: Multigateway status reports
« Reply #74 on: April 07, 2014, 09:30:35 am »

i dont know why but i keep saying ass-toshis in my head instead of asset-oshis! :L
Logged

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #75 on: April 07, 2014, 09:38:46 am »

I've tried to track James posts on this from BCT and here and summarise what the multi-sig gateway looks like so that I can understand it so comments on whether you think i've got it right would be most welcome because I don't want to waste James time to explain his genius to mere mortals like me.

This is my first attempt at it, I will continue this to further my own understanding but if useful to others I will continue to share.

https://dl.dropboxusercontent.com/u/53047281/multi-sig.doc

BR
Ian

PS I've also got a doc in progress on NodeCoin and NXT mixer but those are harder to decipher from James blogs ;)
You are very close!
With the change to assetoshis, there is no need for milliDOGEs anymore
also, for the pure multigateway functionality, all communication is via AM so there is no need for guardians. I really like your chart though! Maybe I will change how it works to match the chart :)

The guardians are more for the NXTmixer and NXTcash, though all these things tend to intermingle and who knows how it will evolve.

James

P.S. Make sure to checkout the API post, it should help you understand with the added details
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #76 on: April 07, 2014, 09:45:58 am »

The line count is creeping up...
I now have 12 files, just for the multigateway handling. This doesnt count the other 10 for NXTprotocol, nodecoin, etc.

Now that I reached code complete, it was time to totally rewrite it again :)

The need for remote nodes to run the multigateway wasnt exactly planned for and that broke an internal assumption, eg. that each computer running multigateway had access to the coin daemons. Now, there are three cases:

A) Full multigateway
B) Validating nodes, does almost all the Full multigateway does, expect for creating new accounts and sending money
C) Normal nodes, which just track things via AM and other NXT blockchain events

In order not to go insane, I need to make sure the same code is run in A) as in C) for the same function. The worst thing is to have two (or three) versions that do pretty much the same thing, but its in different places. One "harmless" change in one place and not the other and you end up debugging for hours.

So, this trigged a major rewrite. The good thing is that the low level code is porting intact, just the messy higher level code that ties it all together. Once all the redundancies have been squeezed out of the code, it becomes much harder for any bugs to remain hidden

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

chanc3r

  • Hero Member
  • *****
  • Karma: +124/-50
  • Offline Offline
  • Posts: 1019
  • NXTInspect
    • View Profile
Re: Multigateway status reports
« Reply #77 on: April 07, 2014, 10:57:59 am »


You are very close!
With the change to assetoshis, there is no need for milliDOGEs anymore
also, for the pure multigateway functionality, all communication is via AM so there is no need for guardians. I really like your chart though! Maybe I will change how it works to match the chart :)

The guardians are more for the NXTmixer and NXTcash, though all these things tend to intermingle and who knows how it will evolve.

James

P.S. Make sure to checkout the API post, it should help you understand with the added details

Hi James,

thanks for the reply and you like my picture :)
I thought I'd removed the fractions-workaround stuff now we have fractions.... I'll amend.
I've found the API thread on this board and that can will help to add / correct some detail.

I'd left the guardian server in because of this in the Infrastructure board request..

Quote from: jl777
I might be able to make a "thin guardian server" that could be run on people's NXT nodes. This would eliminate or reduce the need for a large number of guardian servers and still get a similar (if not better) effect.

So guardians are definitely gone for Multi-Sig-Gateway?

So I can get this straight (and partly with my inf-com hat on) how does a client connect to a gateway server ?
thus how do we ensure they all don't select the same one so that all the wallets don't end up on the same server pair?

Finally I like to add some state-transitions to show the flows between client / gateway-server and xCoind-server, as I think this would be useful, is there anywhere I can do this other than reading your code?
If that's the only way, as I can easily read 'C' is there anyway I can get the relevant code fragments?

Thanks,
Ian
Logged
NXT: 29996814460165 (NXT-JTA7-B2QR-8BFC-2V222)
@imrimr @NXTinspect

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #78 on: April 07, 2014, 12:25:10 pm »


You are very close!
With the change to assetoshis, there is no need for milliDOGEs anymore
also, for the pure multigateway functionality, all communication is via AM so there is no need for guardians. I really like your chart though! Maybe I will change how it works to match the chart :)

The guardians are more for the NXTmixer and NXTcash, though all these things tend to intermingle and who knows how it will evolve.

James

P.S. Make sure to checkout the API post, it should help you understand with the added details

Hi James,

thanks for the reply and you like my picture :)
I thought I'd removed the fractions-workaround stuff now we have fractions.... I'll amend.
I've found the API thread on this board and that can will help to add / correct some detail.

I'd left the guardian server in because of this in the Infrastructure board request..

Quote from: jl777
I might be able to make a "thin guardian server" that could be run on people's NXT nodes. This would eliminate or reduce the need for a large number of guardian servers and still get a similar (if not better) effect.

So guardians are definitely gone for Multi-Sig-Gateway?

So I can get this straight (and partly with my inf-com hat on) how does a client connect to a gateway server ?
thus how do we ensure they all don't select the same one so that all the wallets don't end up on the same server pair?

Finally I like to add some state-transitions to show the flows between client / gateway-server and xCoind-server, as I think this would be useful, is there anywhere I can do this other than reading your code?
If that's the only way, as I can easily read 'C' is there anyway I can get the relevant code fragments?

Thanks,
Ian
guardians aren't totally gone, but I cant say more without leaking info about the secret project...

Most of the gateway <-> client communication is done via AM, which can be generated via the websockets page.

If you can read C, then maybe you can do a source review? Its grown to ~5000 lines and its hard to see the obvious bugs with code you've been looking at for a long time.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

nCtrl

  • Sr. Member
  • ****
  • Karma: +31/-0
  • Offline Offline
  • Posts: 254
    • View Profile
Re: Multigateway status reports
« Reply #79 on: April 07, 2014, 12:29:19 pm »

I can do that as well as long as you let me know where I can find it. (I am on the phone, sorry if it is in plain sight)
« Last Edit: April 07, 2014, 12:30:50 pm by nCtrl »
Logged
Pages: 1 2 3 [4] 5 6 ... 11  All
 

elective-stereophonic
elective-stereophonic
assembly
assembly