elective-stereophonic
elective-stereophonic
Multigateway status reports singapore
Please login or register.

Login with username, password and session length
Advanced search  

News:

Latest Stable Nxt Client: Nxt 1.12.2

Pages: 1 [2] 3 4 ... 11  All

Author Topic: Multigateway status reports  (Read 39672 times)

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #20 on: March 28, 2014, 10:15:14 am »

I am sure these "daily reports" take away a chunck of your time, that maybe could better used but damn they are interesting, the difficulties you face, how you solve them etc. If you can, keep doing it, it is really interesting.
Only takes a few minutes, usually while I am waiting for some tests. Also, it actually helps me by writing stuff down.

Currently, I am getting all the little bugs out of the latest rev. A bit of a hiccup by changing to bitcoind_RPC as it is returning an extra layer of JSON, meaning the data coming from RPC is different than from just running bitcoind. Nothing structural, just have to get everything parsing properly. So far, no roadblocks seen, just tedious work

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #21 on: March 28, 2014, 11:29:17 pm »

Finally got bitcoind_RPC to properly submit createmultisig commands. So, about to get a stream of raw blocks to parse into my internal data structures.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #22 on: March 29, 2014, 04:01:38 am »

For people who complain about NXT not being finished, my feeling is that NXT API is much better than bitcoind as far as stability and speed goes.
At least with dogecoind, there are many RPC calls that take over a second! This is not good at all considering it is running on the same server...
I had thought there was a bug in my code looping a zillion times. Nope, just slow doge

Why the bitcoind JSON parameters are so finicky and inconsistent, it boggles the mind. Sometimes you can put "" around the argument, sometimes you must use them and other times you must not. Doesnt seem to be much rhyme or reason, just need to fiddle with each and every call to make sure it goes through.

So, now I am parsing every block at blazing speeds of 1 block per second. 99.9999% of the time waiting for the slow dogge. I think there must be some sort of curl setting I am not doing right though,it cant be this slow. It will take forever to sync up and I need it to sync up before I can test my code and find bugs.

So many issues not related to my code with this project. Everything goes in slow motion...

James

Edit: I take back the bitcoind is slow thing, http://stackoverflow.com/questions/17383089/libcurl-delays-for-1-second-before-uploading-data-command-line-curl-does-not
I got suspicious when the times were all right around 1000 milliseconds, seemed like it was clocked, like from a timeout. Sure enough the URL above had same problem and solution, now it is 1000 times faster! ~1 millisecond per bitcoind call, onto debugging my stuff!!
« Last Edit: March 29, 2014, 04:24:23 am by jl777 »
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #23 on: March 29, 2014, 05:07:21 am »

Now I am going too fast and bitcoind just randomly stops responding for a while. many seconds, "couldnt connect to server" error, even command line usage is broken during this time.

Again, I am not the first to push the performance limits: https://github.com/bitcoin/bitcoin/issues/2889

I guess 1000 commands per second was too good to be true, I will put in a 10 millisecond delay and hope that works

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #24 on: March 29, 2014, 05:53:36 am »

Now I am going too fast and bitcoind just randomly stops responding for a while. many seconds, "couldnt connect to server" error, even command line usage is broken during this time.

Again, I am not the first to push the performance limits: https://github.com/bitcoin/bitcoin/issues/2889

I guess 1000 commands per second was too good to be true, I will put in a 10 millisecond delay and hope that works

James
No reliable way to throttle the speed, other than to really slow it down.
I will just put in a 30 second delay on error, that seems to give the dogecoind enough time to recover from its broken state

It has reprocessed 5000+ blocks without any errors, I even saw the dreaded OP_RETURN which appeared as one of the vouts without any address or value. Triggered a warning message. The crazy thing is it was usually on a transaction with hundreds and hundreds of vouts, so the total size increase was less than 0.5% and this is in the super rare case that it happens. For all of this "bloat" the bitcoin devs are going to war against XCP.

bitcoin protocol now appears to be controlled by a dozen mining pools, some of which are controlled by bitcoin devs!
decentralized? Hmmm, not exactly.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #25 on: March 29, 2014, 06:03:35 am »

Confirmed deposit.0 DOGE.22431119b6aa6696f91393ff6c25ca3032d3093300a341588018dc918489a13a 1.23456788 -> NXT.11445347041779652448

!!!

Much excitement

The deposit I made that the dogecoind couldnt find for whatever reason, was found by brute force scanning all blocks. Now I can finally test the asset transfer code,which if it works means the deposit side is done. With comments being supported soon, I have a nice solution for specifying the withdraw address with each asset transfer. This way, there is no disconnect from whatever address is registered and where it ends up. It also means I wont need to database it either, this removes another point of failure

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

bitcoinpaul

  • Hero Member
  • *****
  • Karma: +590/-590
  • Offline Offline
  • Posts: 3097
  • Karmageddon
    • View Profile
Re: Multigateway status reports
« Reply #26 on: March 29, 2014, 07:13:53 am »

So many wow!
Logged
Like my Avatar? Reply now! NXT-M5JR-2L5Z-CFBP-8X7P3

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #27 on: March 29, 2014, 08:44:52 am »

So many wow!
Its slow going, but I now have all three servers pretty much in sync and even transferring assets. Just need to get the AM packet created properly so the two servers that didnt do the transfer can update their internal state.

A big issue is to be able to totally reconstruct 100% of multigateway state from scratch, by scanning all the different blockchains and have it exactly match on all three servers. Its harder than it sounds.

pretty much in the home stretch for DOGE, so tomorrow I can test the other coins individually and then all at the same time. So, beta testers, get ready for deposit testing early next week: BTC, LTC, DOGE and DRK

James

Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

CryptoScalper

  • Full Member
  • ***
  • Karma: +10/-24
  • Offline Offline
  • Posts: 192
  • Banned!
  • I DO NOT CONSENT
    • View Profile
Re: Multigateway status reports
« Reply #28 on: March 29, 2014, 09:36:51 am »

Confirmed deposit.0 DOGE.22431119b6aa6696f91393ff6c25ca3032d3093300a341588018dc918489a13a 1.23456788 -> NXT.11445347041779652448

!!!

Much excitement

James

BAM!    ;D


By the way...  as I also mentioned over at BTT in regard to nexern's latest API efforts...  I would like to recommend that you incorporate MetaTrader 4 (MT4)...  please not MT5  :-[...  as the official charting software of your Multigateway.  Don't let all that data from your servers go to waste!

MT4 comes with numerous indicators as it is...  I mean true Technical Analysis (TA) at the palm of your hands...  but the most interesting part is that it is fairly easy for a trader to code his/her own Custom Indicators using MT4's Algorithmic Language MQL4Trust me...  you can't beat it!    ;)

As you probably already know...  MT4 is widely distributed free by most Forex brokerages...  so using it will make your Multigateway (not to mention Nxt itself) very attractive to people with a better understanding of currency trading in general.
« Last Edit: March 29, 2014, 09:40:11 am by smaragda »
Logged
"A world with the money can not be perfect." - BCNext

Mistafreeze

  • Full Member
  • ***
  • Karma: +18/-3
  • Offline Offline
  • Posts: 199
  • I <3 Cheese
    • View Profile
Re: Multigateway status reports
« Reply #29 on: March 29, 2014, 01:54:18 pm »

This truly is so far beyond fantastic!!
Logged
Donate to my Beerfund NXT-L4WV-ZF8P-8X54-D6XML

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #30 on: March 30, 2014, 01:55:13 am »

ran into a nasty race condition bug where a deposit can get credited and transferred, but not seen in time in all cases
need to put some locks and queues and make sure the dataflows avoid most of the problems to minimize the race condition in the first case, but of course have to properly handle it.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #31 on: March 30, 2014, 01:56:43 am »

Confirmed deposit.0 DOGE.22431119b6aa6696f91393ff6c25ca3032d3093300a341588018dc918489a13a 1.23456788 -> NXT.11445347041779652448

!!!

Much excitement

James

BAM!    ;D


By the way...  as I also mentioned over at BTT in regard to nexern's latest API efforts...  I would like to recommend that you incorporate MetaTrader 4 (MT4)...  please not MT5  :-[...  as the official charting software of your Multigateway.  Don't let all that data from your servers go to waste!

MT4 comes with numerous indicators as it is...  I mean true Technical Analysis (TA) at the palm of your hands...  but the most interesting part is that it is fairly easy for a trader to code his/her own Custom Indicators using MT4's Algorithmic Language MQL4Trust me...  you can't beat it!    ;)

As you probably already know...  MT4 is widely distributed free by most Forex brokerages...  so using it will make your Multigateway (not to mention Nxt itself) very attractive to people with a better understanding of currency trading in general.
My understanding is that it is pretty expensive to get an MT4 license to run. However, I can think of ways of implementing a simplified MT4 on top of NXT,but that is for May
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

salsacz

  • Hero Member
  • *****
  • Karma: +239/-67
  • Offline Offline
  • Posts: 1762
  • NXT-R67P-6BZ2-XWAK-8RHZR
    • View Profile
Re: Multigateway status reports
« Reply #32 on: March 30, 2014, 03:23:58 am »

I don't know, where this belongs to: http://www.slideshare.net/mids106/ethereum-write-your-own-contracts - maybe here.
Logged
Holidays are (almost) over, check more news at: Nxt Academy

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #33 on: March 30, 2014, 05:46:14 am »

Finally got the race condition solved, I think
Now adding sanity checks on total and user balances, eg. lets not transfer more assets to user accts than the user has deposited, and make sure total assets transfers is never more than total deposited!

Multigateway will be partially hosted on mynxtcoin.org
I have feelers out to a couple more possible hosts, hopefully will get the required three soon

Now that all the ticky tack bugs seem to be fixed and things are actually doing what they are supposed to, its time to write new bugs :)
Onto withdraws as I am not in the mood for changing to other coins, DOGE all the way to feature completeness!

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

CryptoScalper

  • Full Member
  • ***
  • Karma: +10/-24
  • Offline Offline
  • Posts: 192
  • Banned!
  • I DO NOT CONSENT
    • View Profile
Re: Multigateway status reports
« Reply #34 on: March 30, 2014, 06:38:07 am »

My understanding is that it is pretty expensive to get an MT4 license to run. However, I can think of ways of implementing a simplified MT4 on top of NXT,but that is for May

I am afraid you are correct...  it seems I confused all this free distribution of MT4 by Forex brokerages with the actual licensing   :-[...  from a brief search I saw figures up to $100,000 being thrown around.

Can't wait to see what you implement in May...  in June...  in July...  in August...     ;)
Logged
"A world with the money can not be perfect." - BCNext

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #35 on: March 30, 2014, 09:34:01 am »

Well it feels like toward the end of doing a jigsaw puzzle. Still tricky, but the pieces keep falling into place. Its a good indication that my overall design is solving the problem at hand. It is always much easier the second time around, so the proof of concept version has been very useful, even though almost none of the code has survived the rewrite

Got the low level part of withdraw coded, so looking forward to the painstaking debugging process of getting every syntax perfect so bitcoind does gag. OK, I'm kidding about that. debugging is like hitting your head against the wall, over and over again. The only good thing about it is when it is done.

I am not sure the limits of what the various networks treat as standard transactions, so I need to get the send money part coded so I can characterize that. After I get it debugged tomorrow and find out what the restrictions are, I can map out a plan for managing all of the multisig accts. They start out as deposit accts, but as soon as the NXT asset transfer is done, they are put into a virtual unspent pool, which all the withdraws can use. I have a feeling I wont be able to do more than one multisig source account per withdraw, which isnt a problem if there is an account with a big enough balance. Problem is, I will be making sure there are no big balances, so there could well be multiple transactions needed per withdraw. All depends on each coins miners and what transactions they accept.

Until I know exactly what I need to do, I cant predict how long it will take. Having to write a block level parser delayed things by a couple days, but I am still kind of on schedule as deposits appear to work, at least in one case.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #36 on: March 31, 2014, 07:23:52 am »

Slow progress
It was taking so long to run each test as I had to reprocess a weeks worth of DOGE logs and dogecoind keeps stalling
so did a bunch of code cleanup and got a test framework that allows for much faster turnaround
currently testing the threeway synchronizing of the exact rawtransaction for a hardcoded withdraw. This is finding all sorts of small issues that just need to be fixed before I can even get a withdraw sent. So still not sure how many inputs I can use and stay as standard tx

Still left to do:
a) generating withdraw request from asset xfer, the hard part is to make sure to do it once and only once.
b) broadcasting a moneysent AM and using that to prevent sending money more than once
c) sanity testing with each of the supported coins
d) manually generate testing deposit addresses and <beta release>

e) google authenticator
f) some sort of automated account balancer to prevent any super large acct
g) instrumentation to allow for easy external tracking of balances
h) fix bugs

i) code review
j) release candidate

James
« Last Edit: March 31, 2014, 07:39:09 am by jl777 »
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #37 on: March 31, 2014, 09:34:34 am »

Took all day, but got a synchronized withdrawal signed and submitted. It helps to use the right command "sendrawtransaction" instead of "submitrawtransaction"!
Must be getting tired...
I got txid from dogecoind, but it is not appearing on the network.
probably need to pay the txfee
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: Multigateway status reports
« Reply #38 on: March 31, 2014, 10:12:43 am »

Just as I was waiting for testing the withdraw where i paid the txfee, I lost contact with all my servers :(
since its already tomorrow, i guess its a sign to call it a day and hope the servers are back when I wake up
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

bitcoinpaul

  • Hero Member
  • *****
  • Karma: +590/-590
  • Offline Offline
  • Posts: 3097
  • Karmageddon
    • View Profile
Re: Multigateway status reports
« Reply #39 on: March 31, 2014, 10:13:39 am »

Yeah, get some sleep.

See you in 3 hours.
Logged
Like my Avatar? Reply now! NXT-M5JR-2L5Z-CFBP-8X7P3
Pages: 1 [2] 3 4 ... 11  All
 

elective-stereophonic
elective-stereophonic
assembly
assembly