elective-stereophonic
elective-stereophonic
[ALPHA T10] InstantDEX getting close! Ubuntu pNXTd + pNXTlite for Win32 + Mac singapore
Please login or register.

Login with username, password and session length
Advanced search  

News:

Latest Stable Nxt Client: Nxt 1.12.2

Pages: 1 2 [3] 4  All

Author Topic: [ALPHA T10] InstantDEX getting close! Ubuntu pNXTd + pNXTlite for Win32 + Mac  (Read 12863 times)

valarmg

  • Hero Member
  • *****
  • Karma: +178/-57
  • Offline Offline
  • Posts: 1766
    • View Profile
Re: [ALPHA T10] Ubuntu pNXTd + pNXTlite for Win32 + Mac
« Reply #40 on: July 15, 2014, 11:16:32 pm »

Who are you? What have you done with valarmg?
Are you the satoshi gremlins using brain control?

:)

When encountering a really difficult problem, find an easier one. Intuitively the forced dual pegging was cool, but not really possible to decentralize, could get close with some contortions, but all solved with a decentralized free floating currency. The BTCD community is welcoming this with big excitement and the coin's founder is fully supporting. I have found that the existing cryptonote coins are already set on their path and not really open to anything as radical as grafting on a realtime exchange, multigateways, sports betting, etc. into their coin.

So now, intuitively it feels like a solvable problem, though still with challenges ahead. Thanks for the critical feedback, it really helped to make a better solution possible. I also think all the BTCD guys will be really happy, few of them understand just how important the free floating pNXTd currency will be.

James

Haha, to prove I'm not critical just for the sake of it I have to agree with some ideas!

Just thought that you finally found something useful for PoW miners to do. They have plenty of hardware so are good candidates to set up and run Nxt nodes.

Just thinking that since the decentralized solution works so well, maybe you don't need the second centralized one. What you could do is organize a market-maker for BTCD/NXT transactions. This way you can ensure that the spread is low and liquidity is high. As time goes on and the anonymity function becomes more heavily used, then the market will naturally take care of it.
Logged
NXT-CSED-4PK5-AR4V-6UB5V

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: [ALPHA T10] Ubuntu pNXTd + pNXTlite for Win32 + Mac
« Reply #41 on: July 15, 2014, 11:22:12 pm »

Who are you? What have you done with valarmg?
Are you the satoshi gremlins using brain control?

:)

When encountering a really difficult problem, find an easier one. Intuitively the forced dual pegging was cool, but not really possible to decentralize, could get close with some contortions, but all solved with a decentralized free floating currency. The BTCD community is welcoming this with big excitement and the coin's founder is fully supporting. I have found that the existing cryptonote coins are already set on their path and not really open to anything as radical as grafting on a realtime exchange, multigateways, sports betting, etc. into their coin.

So now, intuitively it feels like a solvable problem, though still with challenges ahead. Thanks for the critical feedback, it really helped to make a better solution possible. I also think all the BTCD guys will be really happy, few of them understand just how important the free floating pNXTd currency will be.

James

Haha, to prove I'm not critical just for the sake of it I have to agree with some ideas!

Just thought that you finally found something useful for PoW miners to do. They have plenty of hardware so are good candidates to set up and run Nxt nodes.

Just thinking that since the decentralized solution works so well, maybe you don't need the second centralized one. What you could do is organize a market-maker for BTCD/NXT transactions. This way you can ensure that the spread is low and liquidity is high. As time goes on and the anonymity function becomes more heavily used, then the market will naturally take care of it.
The pegged transport might only be needed until the "market will naturally take care of it" happens.
By the way ripple has that philosophy and millions of dollars and years later, they are still waiting
I dont like to wait, so I will provide the pegged transport, which interestingly should create the market size needed and maximizes the chance for "market will naturally take care of it" to happen

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: [ALPHA T10] Ubuntu pNXTd + pNXTlite for Win32 + Mac
« Reply #42 on: July 16, 2014, 01:47:20 am »

BTCD just passed a petahash of mining!
I think that qualifies as secure and provides a nice currency to use as pNXTd transport

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

VanBreuk

  • Hero Member
  • *****
  • Karma: +362/-19
  • Offline Offline
  • Posts: 2772
    • View Profile
Re: [ALPHA T10] Ubuntu pNXTd + pNXTlite for Win32 + Mac
« Reply #43 on: July 16, 2014, 08:36:17 am »

*adds BTCD to regular bookmarks*

error -5 is time sync error
have you done "ntp" recently?

I did update against pool.ntp.org and the error -5 was replaced with result:null. No luck so far...
Logged
GPG Fingerprint: B020 D1C1 F289 3B2C 3577  9EAD 455D D175 5913 C7F1

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: [ALPHA T10] Ubuntu pNXTd + pNXTlite for Win32 + Mac
« Reply #44 on: July 16, 2014, 08:57:25 am »

*adds BTCD to regular bookmarks*

error -5 is time sync error
have you done "ntp" recently?

I did update against pool.ntp.org and the error -5 was replaced with result:null. No luck so far...
I think now the previous version might work for you, but I introduced some sort of memory bug in the meantime.
pretty nasty and too tired to fix tonight, but at least I am halfway done getting NXTatomic into pNXT, next couple days will be kind of busy so probably weekend before we can see some actual InstantDEX trades
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

valarmg

  • Hero Member
  • *****
  • Karma: +178/-57
  • Offline Offline
  • Posts: 1766
    • View Profile
Re: [ALPHA T10] Ubuntu pNXTd + pNXTlite for Win32 + Mac
« Reply #45 on: July 16, 2014, 12:01:02 pm »

The pegged transport might only be needed until the "market will naturally take care of it" happens.
By the way ripple has that philosophy and millions of dollars and years later, they are still waiting
I dont like to wait, so I will provide the pegged transport, which interestingly should create the market size needed and maximizes the chance for "market will naturally take care of it" to happen

James
The more I think about it, the more I like this dual transport idea. While the service is starting and underused, there's no big deal using a centralized gateway. It's only when it becomes big that the problems of the centralized solution arises, in which case the decentralized transport layer can easily take over. It's basically a bootstrapping phase to get the system up and running.

I would just say to make the centralized solution as simple as possible. Just 1:1 peg on pNXT tokens which are used for privateSend. Put more advanced features (like the ability to store and directly transact with the anonymous currency) in the decentralized option, so it's easy to remove the the centralized gateway without hurting the overall system at a later date.




Logged
NXT-CSED-4PK5-AR4V-6UB5V

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: [ALPHA T10] Ubuntu pNXTd + pNXTlite for Win32 + Mac
« Reply #46 on: July 16, 2014, 06:18:35 pm »

The pegged transport might only be needed until the "market will naturally take care of it" happens.
By the way ripple has that philosophy and millions of dollars and years later, they are still waiting
I dont like to wait, so I will provide the pegged transport, which interestingly should create the market size needed and maximizes the chance for "market will naturally take care of it" to happen

James
The more I think about it, the more I like this dual transport idea. While the service is starting and underused, there's no big deal using a centralized gateway. It's only when it becomes big that the problems of the centralized solution arises, in which case the decentralized transport layer can easily take over. It's basically a bootstrapping phase to get the system up and running.

I would just say to make the centralized solution as simple as possible. Just 1:1 peg on pNXT tokens which are used for privateSend. Put more advanced features (like the ability to store and directly transact with the anonymous currency) in the decentralized option, so it's easy to remove the the centralized gateway without hurting the overall system at a later date.
Yes, bootstrapping is the right approach!
It seems the future is bright for BitcoinDark :)
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: [ALPHA T10] Ubuntu pNXTd + pNXTlite for Win32 + Mac
« Reply #47 on: July 19, 2014, 10:22:55 am »

Been stuck on a hard to find bug for a few days. Of course, also did the JLH thing, and BTCD and MGW, so its not like I was idling :)

Anyway, I can finally get back to completing the second half of order matching for InstantDEX.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: [ALPHA T10] Ubuntu pNXTd + pNXTlite for Win32 + Mac
« Reply #48 on: July 22, 2014, 07:25:53 pm »

Current status of basic privacyServer operations:

client -> privacyServer selection (non-encrypted) - works
client -> privacyServer encrypted UDP channel - works
client -> privacyServer encrypted TCP channel - works
privacyServer -> client non-encrypted handshake - works
privacyServer -> client encrypted UDP channel - works
privacyServer -> client encrypted TCP channel - untested (actually currently this path is not used)
client -> privacyServer -> another client via two layer onion routing - works

I was thinking about each privacyServer keeping a directory of clients that it is responsible for. Just the public acct # and the public key. However now I am leaning toward just broadcasting any client -> client comms that are not connected via the same privacyServer. This would add bandwidth, but it eliminates a potential attack vector. I cant think of how an attacker can use the directory of public addresses and keys, but if there is no directory, then I think it is one less piece of data that is available. The less data to be correlated, the better.

So the logic will be:
client selects privacyServer. Now most commands are routed through the privacyServer. API calls like getOrderbooks, placeBid, etc. These just go to the privacyServer and it handles it. OK, so that's the easy stuff. The hard part is what to do when you want to communicate with someone else. Let us assume that the privacyServer can magically know how to communicate to everyone, but we dont want other people to be able to snoop and find out that you are.

Is this even possible? Well, this is where the fact that we trust the privacyServer comes in really handy.

Say A wants to send a message to B without letting anybody except the privacyServer know and the only thing the privacyServer will know is that A send something to B, but even the privacyServer wont know WHAT was sent.

A first makes the message then tokenizes it with a timestamp. This both proves that A made the message and at what time. It prevents not only spoofing, but also replay attacks. A replay attack is where the attacker makes a copy of a valid message and rebroadcasts it. Maybe it doesnt do anything bad, but it could certainly confuse things, so I like to timestamp, tokenize and then encrypt. OK, so here is one problem I need B's public key to encrypt the message to him. So I need to magically get that somehow, maybe from a query to the server or simply processing broadcasts of the public keys. Either with pros/cons, I will decide later on specifics for that.

OK, so now we have a time validated, account validated and encrypted message to B. Only the fact that it is addressed to B is possible to be determined by getting this packet.

So can we send that? Well, not quite. What if the attacker has taps into the major internet routers? You dont want to be sending stuff in the clear (unencrypted). That would allow the attacker to know that A is sending something to B. So, we add another layer of encryption by re-encrypting the packet so that only the privacyServer can decode it. Now the attacker that is monitoring all internet packets will only see that you sent something to the privacyServer, but you send everything through the privacyServer, so it really doesnt divulge much information.

If you are with me so far, we now look at what the privacyServer gets. It is an encrypted packet that it is able to decode and it sees that what it gets is another encrypted packet that is addressed to B. So, it then "just" sends it to the privacyServer that B selected, or broadcasts it to all nodes. Now anybody that intercepts this packet will only know that it came from the privacyServer and not A.

So, B finally gets the packet, is able to decrypt it and gets the message from A.

Now that I write this, I do think it is better for the privacyServers to keep a directory of all accts/publickeys and then each client can query the privacyServer for the public key. There is much less info divulged by doing this as each acct can broadcast to everyone its public key when it comes online. So there is no correlation possible and the queries about B's publickey is done under encryption, so no info leaked.

It helps me think about things by writing these long texts. I hope I didnt bore you with wall of text. If anybody can see ANY flaws in this logic, please let me know ASAP. just describing this has not been easy, imagine coding and debugging it!
I hope you can see that once there is this encryption network in place, then all sorts of things become possible, all while remaining private. Businesses can conduct business using a public address, but its privacyServer will route everything to private addresses that only it knows about. This will allow businesses to conduct business without letting their competitors know all their income and expenses and even from whom.
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: [ALPHA T10] Ubuntu pNXTd + pNXTlite for Win32 + Mac
« Reply #49 on: July 23, 2014, 01:26:31 am »

For those that want to follow my progress in detail on InstantDEX, you can follow it from the BTCD thread: https://bitcointalk.org/index.php?topic=684090.new#new

For those worried that now I am core dev for BTCD that I wont have time for NXT, dont worry!
There is 95% overlap between my NXT works and BTCD.
BTCD has cleared resonated with the bitcoin community and traders, actually just a bit behind LTC in trading volumes. So when people find out that it is NXT that is enabling BTCD to be so cool, well, I expect quite a significant increase in NXT popularity

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

Ezravdb

  • Hero Member
  • *****
  • Karma: +42/-12
  • Offline Offline
  • Posts: 910
    • View Profile
Re: [ALPHA T10] Ubuntu pNXTd + pNXTlite for Win32 + Mac
« Reply #50 on: July 23, 2014, 02:02:57 am »

For those that want to follow my progress in detail on InstantDEX, you can follow it from the BTCD thread: https://bitcointalk.org/index.php?topic=684090.new#new

For those worried that now I am core dev for BTCD that I wont have time for NXT, dont worry!
There is 95% overlap between my NXT works and BTCD.
BTCD has cleared resonated with the bitcoin community and traders, actually just a bit behind LTC in trading volumes. So when people find out that it is NXT that is enabling BTCD to be so cool, well, I expect quite a significant increase in NXT popularity

James

much respect jl..
Logged
head

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile
Re: [ALPHA T10] Ubuntu pNXTd + pNXTlite for Win32 + Mac
« Reply #51 on: July 24, 2014, 05:23:23 am »

For those of you worried about all the data the privacyServer will collect on you, stop worrying

other than your IP address and what accts you are in contact with, there is really not much else the privacyServer gets to see. And as discussed before there are ways to minimize or eliminate any downside with both of these (proxy server and extra layer of onion routing)

So the privacyServer gets to see stuff like this:

ONIONIZE: np.0x7ff740002e40 NXT.8989816935121514892 8989816935121514892 pubkey.6eddccdbaf6f08bf encode len.809 -> new len.865 + 42 = 907
sendmessage (<encrypted>) len.809 to 8989816935121514892

sendmessage ([{"status":"1700913285397123524 sends direct encrypted sendmessage.(<encrypted>) [<encrypted>] to 8989816935121514892 pending"},{"token":"vb5lpi57ea2pb4cvjn7g1i2ffrso4j7p7ban3vfmhv7ftcbp7qjak4et74bqje81s7krug8htg3nha3 o0n4aahulm9kt05rcm581qg00uhug2mn5443av4vn88preune1apsm5hjoek7t6arrpgj7oma5m9kir i1"}]) len.303 to 8989816935121514892

The destination account number and size of packet. It sure makes debugging super hard, as I have no way of seeing what data the server is getting and so until all the encryption/decryption was flawless, it was very much stumbling in the dark. [yes, observant readers notice now the encryption/decryption is flawless at least for two layer loopback onion routing]

But this sort of details are not why I am posting. I wanted to announce that the first matched trade was completed!

I will skip the dozen steps that led to:

{"result":"tradecompleted","txid":"6544081069274036129","signedtx":"0201aaa63e01d00225c5fed2690701cf06f267e7c227b1a3c0dfa9c6fc3cdb593b3af6f16d65302 f8c71b555df3ec27c000000000000000000e1f50500000000000000000000000000000000000000 0000000000000000000000000000000000ff1355c269aef4947f6a3842931f8d8e85b01a9abe8c7 9643dc61a3aeca83e0def1c3d96818b0ef6a727eed397cb4803c3af78dde94fa209bd914790a0aa 12ca444a53bf9828e869e80300000000000031007b22617373657442223a2231363339323939383 439333238343339353338222c2271747942223a2231303030303030227d","othertxid":"7139578165401755004"}

These two transactions are on the actual NXT testnet and they decode as follows:

{
    "fullHash": "a1f76f7b0f3fd15a1da4113729f6ec855ddf27b9175f9bfd081279af67951e6b",
    "confirmations": 6,
    "transaction": "6544081069274036129",
    "block": "2448711563708809301",
    "attachment": {
        "asset": "7631394205089352260",
        "quantityQNT": "1000",
        "comment": "{\"assetB\":\"1639299849328439538\",\"qtyB\":\"1000000\"}"
    },
    "recipient": "8989816935121514892",
    "sender": "8989816935121514892",
    "timestamp": 20883114,
    "height": 163875,
}

{
    "referencedTransactionFullHash": "a1f76f7b0f3fd15a1da4113729f6ec855ddf27b9175f9bfd081279af67951e6b",
    "confirmations": 6,
    "transaction": "7139578165401755004",
    "block": "7001348228112354675",
    "attachment": {
        "asset": "1639299849328439538",
        "quantityQNT": "1000000",
        "comment": "{\"assetA\":\"7631394205089352260\",\"qtyA\":\"1000\"}"
     },
    "recipient": "8989816935121514892",
    "sender": "8989816935121514892",
    "timestamp": 20883114,
    "height": 163876,
}

You will notice that these two transactions are the opposite of each other. The comment field has what each transaction wants in return. Also key is that the second transaction has a referencedTransactionFullHash, which matches the fullHash of the first tx. This is how the two are linked, so the second transaction would wait until the first transaction was seen on the blockchain. In order to give assurance to the other party that the transaction was indeed submitted, not only is the txid sent back, but also the signed tx bytes. this way both parties can broadcast the transaction that they like, eg. the one where they get stuff. I am automating all this so it all just works, but the worst an attacker can do by changing the code is to prevent the broadcast of the first tx or the second tx. If he stops the first one from going out, the second one never happens and will expire. If he stops the second from from going out, well why would he, thats the transaction that gives him his half, so in order to collect on that he has to broadcast his tx.

What does all this mean???

It means InstantDEX can now matchup exact trades and get them to trade nearly atomically. There is a remote possibility that if you are trading with the next node that will forge a NXT block that he could do some nasty things, its very complicated and really more a theoretical thing since only accounts with a large balance are likely to forge the next block. [noashh plz note] Still I will add in a check to see if you are trading against the node that will forge next (NXT will allow this to be known) and then simply prevent any trading with that node, so we will sidestep even this theoretical attack.

Now we are getting to higher and higher level code, so what I write is far removed from the brain tangling onion routing and dark packets I cant even see anything in. Next on the list is, let me think,...

OK, I guess next on the list is the infrastructure to be able to support the simple tradebots. Now this might sound pretty simple, but there are actually quite a few different pieces that are needed.

It will probably take me into the weekend to get to where you can command your own tradebot.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

Daedelus

  • Hero Member
  • *****
  • Karma: +230/-12
  • Offline Offline
  • Posts: 3280
    • View Profile

Whoop!  ;D
Logged
NXT: NXT-4CS7-S4N5-PTH5-A8R2Q

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile

Whoop!  ;D
I coded the datafeed input today, so prices from bter, cryptsy, mintpal, poloniex, bittrex and of course NXT AE will all be available. However in order to debug all this, I need to see the data, so I am taking a several day detour into visualization stuff. Good news is by next week we will get near realtime graphic display of prices with some nice colored lines

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile

Well, the debugging of the exchange data gathering went faster than expected. It just sort of worked after a few minor issues.

http://209.126.70.170/exchange_data has a Mac only version that will create an exchanges subdirectory and start gathering data for the pairs in the MGW.conf file which needs to be in your home directory

You will need around 12 hours of data to be able to see good charts, so if you have a Mac, hopefully you can start gathering the data while I finish the display stuff.

Here is a sample MGW.conf file between the "###" lines
###
{
"MAINNET":1, "ORIGBLOCK":"16232361214057123821",
"active":["DOGE"],
"coins":[
{"name":"BTC","conf":"/home/root/.bitcoin/bitcoin.conf","asset":"4551058913252105307","txfee":0.0001,"NXTfee_equiv":0.0001,"rpc":"127.0.0.1:8332","blockheight":300797,"forkblockheight":308539,"minconfirms":3,"estblocktime":600},
{"name":"LTC","conf":"/home/root/.litecoin/litecoin.conf","asset":"2881764795164526882","txfee":0.001,"NXTfee_equiv":0.01,"rpc":"127.0.0.1:9332","blockheight":564980,"forkblockheight":594965,"minconfirms":5,"estblocktime":150},
{"name":"DRK","conf":"/home/root/.darkcoin/darkcoin.conf","asset":"17353118525598940144","txfee":0.01,"NXTfee_equiv":0.01,"rpc":"127.0.0.1:9998","blockheight":49594,"forkblockheight":0,"minconfirms":7,"estblocktime":150},
{"name":"BTCD","useaddmultisig":1,"nohexout":1,"conf":"/home/root/.BitcoinDark/BitcoinDark.conf","asset":"11060861818140490423","txfee":0.0001,"NXTfee_equiv":0.1,"rpc":"127.0.0.1:14632","blockheight":16000,"forkblockheight":0,"minconfirms":3,"estblocktime":60},
{"name":"DOGE","conf":"/home/root/.Dogecoin/dogecoin.conf","asset":"11749590149008849562","txfee":1.0,"NXTfee_equiv":200.0,"rpc":"127.0.0.1:22555","blockheight":250000,"forkblockheight":0,"minconfirms":3,"estblocktime":60}
],
"timezone":"-3",
"bittrex":[["BTCD","BTC"]],"cryptsy":[["BTCD","BTC"],["NXT","BTC"]],"poloniex":[["BTCD","BTC"],["NXT","BTC"]],"bter":[["NXT","BTC"]],"mintpal":[["LTC","BTC"]],"NXT":[["BTC","NXT"]]
}
###
You really only need the last part where the exchanges are listed. For "NXT" as the exchange, only coins specified in the "coins" field will work and the second part needs to be "NXT". For all the other exchanges, you should be able to put in any coin that they trade, though you probable need to put it in the right order to match how they are trading it.

Supported exchanges: "NXT", "bittrex", "cryptsy", "poloniex","bter" and "mintpal"

The time displayed will adjust by the "timezone" specified hours and should take into account daylight savings for USA peoples

The program loads the orderbooks of the specified pairs, no more frequently than once per second to an exchange and additionally no more frequently than once per ten seconds for a specific pair. This should get most all of the price changes without getting you blacklisted by the exchanges for abusing their API. Each time the top of the orderbook changes, it prints out the details and also adds an entry to a binary file.

Tomorrow, I will process this file and start getting some simple charts displayed to verify proper data and then finally I can code some tradebot logic, hopefully on Sunday

James

P.S. Please post any bugs that you find, it is really strange that it just worked the first time for all the exchanges, especially since they all have different syntax, especially mintpal and NXT
« Last Edit: July 26, 2014, 08:28:09 am by jl777 »
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

valarmg

  • Hero Member
  • *****
  • Karma: +178/-57
  • Offline Offline
  • Posts: 1766
    • View Profile

I had an idea, dunno if this has been discussed before.

One of the advantages of this network of servers that will be set up is that a backbone is created for Nxt to run on. However, these servers mightn't have much NXT, and if so, wouldn't be forging blocks that often. So I suggest that a portal be created called, say, NxtLeasing. Anyone running a server points it at this portal and NXT gets leased to them. Anyone who has Nxt and doesn't want to run a node, they just lease to NxtLeasing, and NxtLeasing takes care of distributing the leases evenly among the servers (or maybe the servers are monitored and more lease is distributed to the servers that are always online and have sufficient specs to handle the network load). That way, we can have a centralized place so everyone who wants to lease knows where to go, while keeping the network decentralized since anyone can point their server at the portal and have Nxt leased to them.
Logged
NXT-CSED-4PK5-AR4V-6UB5V

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile

I had an idea, dunno if this has been discussed before.

One of the advantages of this network of servers that will be set up is that a backbone is created for Nxt to run on. However, these servers mightn't have much NXT, and if so, wouldn't be forging blocks that often. So I suggest that a portal be created called, say, NxtLeasing. Anyone running a server points it at this portal and NXT gets leased to them. Anyone who has Nxt and doesn't want to run a node, they just lease to NxtLeasing, and NxtLeasing takes care of distributing the leases evenly among the servers (or maybe the servers are monitored and more lease is distributed to the servers that are always online and have sufficient specs to handle the network load). That way, we can have a centralized place so everyone who wants to lease knows where to go, while keeping the network decentralized since anyone can point their server at the portal and have Nxt leased to them.
I am not so worried about such details. Without high capacity hub and spoke infrastructure, there is no TF. With such an infrastructure, then clever people will solve how to use the capability to enable TF. I am working on getting all the hubs in place as that seems to be something just being left to "organic growth" (chance?)
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile

I got a bit carried away and wrote another 1000 new lines of code...
I just wanted things to be as good as possible, so after I got the relatively easy parsing in of the data that is being logged by yesterday's exchange_data, I had to figure out how to update in realtime the various contract pairs (BTCD/BTC and NXT/BTC are contracts at least from forex terminology stuck in my head).

OK, so making a new thread not so big of a deal, but we have historical data and new data and I dont want to have separate code paths for doing mostly the same thing, eg. updating the internal price data structures.

The we have the issue of multiple exchanges with a separate but clearly related data stream, so each contract pair has one set of price data for each exchange and one that is a combination of all of them.

Now, how to combine the data? Also, how to display the data? I dont like having to keep switching timeframes when looking at prices, so I coded a variable time scale. What that means is that the closer you are to now, the faster the timeframe and the farther away you are the slower. I tried to make a linearly compounding timescale so hopefully it will be a smooth compressing of past data. This way, with one page you can see everything from the last minutes to the last days (and even weeks)

OK, so each exchange will necessarily have a different timescale, at least theoretically, so I have to recalibrate the summary graphs to take this into account.

Separately, a lot of price changes are noise and I want to smooth out as much of that as possible so you can see the important price changes. To do this, I use what are called splines. These are handy little guys that create a smooth curve fitted line to a set of data points. Of course, the raw data needs to be processed to a form appropriate for splining. I didnt get a change to put in noise filters, just using some basic averaging for now. Oh, yes, I also create a BAR for each pixel. my bar is more than just open close high and low, it has 16 total fields including those. This will be handy later when querying the privacyServer for price data, I figured might as well do it right from the beginning.

Using some arbitrary weights I assigned to each field and using a 1/(timedist^2) weighing for each spline reference point, I create what I hope are a nice importance weighted and time synchronized inputs for the splining.

With a spline you can query it and get the value, slope even accel at any given point in time. Of course it is just an interpolation, but it lets you draw smooth lines for the entire period. [It is even possible to extrapolate, but that is best waited until I put in my SVM predictors]. So, of course, now we can much more easily combine all the exchanges data as the spline outputs can be for any time period

And all of this in a relatively clean control flow to automatically generate .jpg files that will deliver the right one based on the HTML on the :7777 page.

Anyway, that's what I coded today and about to debug. Probably will take me till tomorrow to get this debugged, but then it is nice framework for hooking in the simple tradebots. Oh, I also am dynamically merging all orderbooks into a single one so all that bookkeeping stuff is coming along.

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer

AEcharts

  • Newbie
  • *
  • Karma: +1/-0
  • Offline Offline
  • Posts: 13
    • View Profile

What do you think about the charts I have created for the AE. Here is a link to a post I made on the Asset Board: https://nxtforum.org/index.php?topic=3451.0

I was wanting to see if anyone was interested in investing in something like this so I could move forward with it, but I didn't get any replies.

The charts could be made available over an HTTP API. The values of a number of different technical indicators could also be made available in this way also. For example if you wanted to know what the value of the MACD configured at 12 for fast, 26 for slow, 9 for signal, using an EMA was at any given time on 1 hour price chart. Or what the values of the upper, lower, mid bollinger band values are etc. The simple trade bots could then use these values.

I am also wanting to provide other unique information about the time series analysis that isn't often available with other charts. For example what is the correlation, covariance, etc over a moving window between 2 different instruments or contracts as you call them. From there I would like to do even more interesting things, but that would be a start.

Let me know if you are interested in any of this.
Logged
AEcharts: NXT-P7F9-P9DJ-YZXH-42UJW

jl777

  • Hero Member
  • *****
  • Karma: +718/-123
  • Offline Offline
  • Posts: 6170
    • View Profile

What do you think about the charts I have created for the AE. Here is a link to a post I made on the Asset Board: https://nxtforum.org/index.php?topic=3451.0

I was wanting to see if anyone was interested in investing in something like this so I could move forward with it, but I didn't get any replies.

The charts could be made available over an HTTP API. The values of a number of different technical indicators could also be made available in this way also. For example if you wanted to know what the value of the MACD configured at 12 for fast, 26 for slow, 9 for signal, using an EMA was at any given time on 1 hour price chart. Or what the values of the upper, lower, mid bollinger band values are etc. The simple trade bots could then use these values.

I am also wanting to provide other unique information about the time series analysis that isn't often available with other charts. For example what is the correlation, covariance, etc over a moving window between 2 different instruments or contracts as you call them. From there I would like to do even more interesting things, but that would be a start.

Let me know if you are interested in any of this.
I did charts today:



This is just from half a day's data and just the basic, price, slope and slope of slope displayed in green, purple, blue

I dont like having to keep switching timescales, so I have a variable timescale implemented. It starts out at one minute per pixel (rightmost) and gets longer and longer as you go to the left. This way you can see the entire dataset, but still see the most recent price movements.

I also dont like the big delays that standard SMA and even EMA have, you cant see it on this chart as I removed the raw data, but it is essentially random noise away from the green line. Once there is enough data gathered, then it will be possible to do much better.

I am now able to automatically gather data from NXT, bter, poloniex, mintpal, cryptsy and bittrex for any pair they support. All this is integrated into InstantDEX API and I will have API calls so you can retrieve data for specific timeframes. It even automatically updates in realtime as the data comes in, all via the localhost :7777 page

James
Logged
There are over 1000 people in SuperNET slack! http://slackinvite.supernet.org/ automatically sends you an invite

I am just a simple C programmer
Pages: 1 2 [3] 4  All
 

elective-stereophonic
elective-stereophonic
assembly
assembly