Nxt Forum

Nxt Discussion => Nxt Projects => Topic started by: altsheets on August 08, 2015, 01:35:33 pm

Title: chaincountdown - blockchain aware countdown timer with image server
Post by: altsheets on August 08, 2015, 01:35:33 pm
chaincountdown.py v14 (https://nxtforum.org/index.php?topic=9735.msg206995#msg206995)

Time estimation to a target block height of nxt/hz/....

Image rendered at request. Comes with httpserver.
Choose heading, fontsizes, fontface, colors.

Built for for the nxthacks voting, and beyond.

This PNG image is generated automatically, and always up-to-date:

The image style can be adapted to any webpage, like here (in the center):

Placed under my giveback license v05 (http://altsheets.ddns.net/give/). Please give generously. Thx.

Much more infos, e.g. on how to choose better colors, sizes, font, etc:


A Blockchain aware countdown timer !

(links to my nxthacks project "assetgraphs.py")

(links to my "chaincountdown" image server) (http://altsheets.ddns.net:8888)
Title: Re: chaincountdown - blockchain aware countdown timer with image server
Post by: altsheets on August 10, 2015, 09:18:12 am
New much improved version v10:

v10 Fonts & Static pages - detected automatically.

* all fontnames & static files are collected at startup
* new modules: helpers.py, imaging_tests.py
* two new image generators:
* all available fonts - 'allfonts.png' image is re-rendered when you call: python imaging.py
* all fonts with increasing size (to test minimum readable fontsize). Call: python imaging_tests.py
* new static page: errors.html with all errors that are caught

v09: all automatic now:

* at startup, prints folders, username, and current time, good for -->
* logging
* README-serverinstall.txt : starts the webserver not as root but as dedicated user

(http://altsheets.ddns.net:8888/allfonts.png) (http://altsheets.ddns.net:8888)

You choose your own fonts!
(http://altsheets.ddns.net:8888/nxt/495997.png?heading=test%20our%20projects%20and%20vote%20now%20in%20the%20nxthacks:&fontsize=45&headingfontsize=25&color=22ff00&bgcolor=0055ee&font=Furore) (http://altsheets.ddns.net/assetgraphs/)

And *tattaa* the first donation came in! Thx!

Learn more at: http://altsheets.ddns.net:8888

Title: Re: chaincountdown - blockchain aware countdown timer with image server
Post by: altsheets on September 02, 2015, 02:52:00 am
v11: "Conceptual Thunderstorm" - coding in the old programming language "English":

* new architecture.html
* new todo.html
* better errors.html
* github wikiwiki
* github issues
* I started this session (the 3rd after waking up 26 hours ago) in a MINDMAP software.
* What inspired me much ... to still continue, and happily... was this reaction :
   " It works! This is awesome. Tomorrow, i will make it work for ethereum. "
   Thanks! Feedback is a booster ...

v12: ... learning Docker. My very first ...

* ... Dockerfile, and
* DockerHub repository altsheets/chaincountdown
* and a lot of extra learning, because "Windows is different" again :-)
* already have an idea for an even better Dockerfile
   - one which always pulls the newest sourcecode from github. Next time ...
* oh, and actually *lol* ...
   I have won the nxthacks2015 (https://nxtforum.org/index.php?board=199.0) - the voting poll (https://nxtforum.org/index.php?topic=9824.0) of which was the inspiration for this chaincountdown.

v13: Docker magic

* learnt lots more about Docker, powerful stuff ... and applied it:
* see github chaincountdown/docker for these files, and more info on:
  * clonethenrunserver.sh is running inside the container
      * it cleverly clones the newest source code from github, before it starts the CCD server
      * and alternatively, withssh.sh additionally allows for ssh access into the docker container
  * dockerbuild.sh is building an image from the Dockerfile
  * dockercheatsheet.txt = useful docker commands
  * README.md exlaining it all
* autostarting using systemd
   * chaincountdown.service for systemd.
   * instructions how to install it.

ALL of the above, with tons of URLs linking to all the files ...
... in the LIVE VERSION (http://altsheets.ddns.net:8888/#history) of this neat and useful image server.

Title: Re: chaincountdown - blockchain aware countdown timer with image server
Post by: altsheets on January 06, 2016, 03:34:41 pm
version update

v13.5: Block times revisited
  • new blocktime measurements for HZ and NXT,
    see github config.py (https://github.com/altsheets/chaincountdown/blob/master/config.py#L24) and /tools folder (https://github.com/altsheets/chaincountdown/tree/master/tools)
  • NXT block 621,000 = hard fork
    • NXT example pic (at the top of the server page (http://altsheets.ddns.net:8888/)) changed
    • post in nxtforum ... nrs-v1-7-4 (https://nxtforum.org/nrs-releases/nrs-v1-7-4/msg205165/#msg205165)
      - suggestion to use these pics to alert people
  • updated donations, history, links to my other projects
    (bottom of example server page (http://altsheets.ddns.net:8888/#giveback))

ChainCountDown (CCD) image server:
thread nxtforum (https://nxtforum.org/nxt-projects/chaincountdown-blockchain-aware-countdown-timer-with-image-server/)
thread bitcointalk (https://bitcointalk.org/index.php?topic=1147294.0)
github (https://github.com/altsheets/chaincountdown) - shorturl 2020.fm/ccd (http://2020.fm/ccd)
live server (http://altsheets.ddns.net:8888/)


If you like this, show it:   NXT-CMKU-ZQYK-V6CD-9UHF4   - thanks.

Live rendered example image:

http://altsheets.ddns.net:8888/nxt/621000.png?heading=Upgrade%20your%20NXT&hea.... (http://altsheets.ddns.net:8888/nxt/621000.png?heading=Upgrade%20your%20NXT&headingfontsize=40&color=ffffff&bgcolor=476C8E)

Please retweet this:

(http://i.imgur.com/IQT1U0y.png) (https://twitter.com/altsheets/status/684768780451471360)

Title: Re: chaincountdown - blockchain aware countdown timer with image server
Post by: altsheets on January 23, 2016, 11:18:31 pm
EDITED heavily, to avoid repetition below. See this post (https://nxtforum.org/index.php?topic=9735.msg207212#msg207212) for the continuation of this.

chaincountdown v18
Motivation, why all this --> A user had complained that my image server 'ChainCountDown (http://altsheets.ddns.net:8888/)' was broken ... but it actually was jxnt.org which was down - which is the server I had been using for API queries (of 'getBlockchainStatus' --> blockNumber). New solution: Whenever an API lookup is failing, the next server is asked. Only if all nodes are failing, my server is in trouble. But now server names are hardcoded in config.py - so I wanted to select servers which may last a while. Choosing by looking at their domain names.


Domain lookup for NXT nodes and HZ nodes
PeerExplorer for HZ - updated!

Code: [Select]
mkdir -p testing; cd testing; rm peerMagic.py* -f
apt-get -y install wget python
wget https://raw.githubusercontent.com/altsheets/chaincountdown/master/tools/peerMagic.py
python peerMagic.py
Or on Windows: install Python 2.7 (https://www.python.org/downloads/), then download (https://raw.githubusercontent.com/altsheets/chaincountdown/master/tools/peerMagic.py) & doubleclick peerMagic.py

You can
* study the source code (https://github.com/altsheets/chaincountdown/blob/master/tools/peerMagic.py)
* look at the NXT results (https://github.com/altsheets/chaincountdown/blob/master/tools/peerMagic_NXT.txt)
* and the -really interesting- HZ results (https://github.com/altsheets/chaincountdown/blob/master/tools/peerMagic_HZ.txt),
* and see the network analysis results below.



... I could use www.peerexplorer.com results, so the main work was a multi-threaded DNS lookup.

Plus meaningful sorting mechanisms:
* The domain names are sorted back to front, with TLD as the most significant split part.
* The IP addresses are in increasing order, with most significant the left-most byte.

For 30 of the 46 IPs with open nxt API ... I got domain names:

Code: [Select]                       94-226-61-134.access.telenet.be                           d515311F0.static.telenet.be                                   ps12.bitcoindark.ca                                   ps14.bitcoindark.ca                                         ai.lasthop.ca   ec2-52-28-97-208.eu-central-1.compute.amazonaws.com   ec2-52-29-122-21.eu-central-1.compute.amazonaws.com                    192-3-196-10-host.colocrossing.com                                  nxt.cybermailing.com                                vader.cybermailing.com                                                111-193-46-198.openvirtuals.com                     cpe-76-176-202-107.san.res.rr.com                                  host4.topserver4.com                               astra2991.server4you.de                                  euve35997.vserver.de            static.                                  127.ip-176-31-167.eu                                   daemon.bevolved.net                                     multi.djaeger.net                                 25.ip-167-114-113.net                                250.ip-167-114-113.net                                 27.ip-167-114-113.net                                  humanoide.thican.net                                    megaman.thican.net    cpc80669-stap13-2-0-cust118.12-2.cable.virginm.net                                 h6113.upc-h.chello.nl                               abctc.vps.nxtcrypto.org                           jefdiesel.vps.nxtcrypto.org                                                               nxt.sx                                             qob.pp.ua

These are all IPs with open API!


For Horizon HZ

... I could not query any peerexplorer.com server, so I developed my own crawler:

(1) ask 'woll-e.net' for 'getPeers' ... answer is 659 APs (address+port, or IP)
(2) enqueue them all, and start 150 worker threads
(3) in each thread ... dequeue one AP, add it to 'apDone'.
(4) if an IP answers to a 'getTime' query on 7776, save it to 'openAPI'.
(5) if AP is answering on peer node port (usually 7774) to 'getInfo', then query 'getPeers' - then enqueue all its (yet-unvisited) peers.
(6) store all results in lists, do cool infoline printing ...
(7) keep on working through the queue ... until it is finally empty:

Ready. After checking 669 APs, on two ports each (and in ~20 seconds only!) ...:

187 HZ nodes, and
63 with open API (answering on port 7776).


Of 63 nodes with open API, 44 had a domain name.
To see all details, scroll down in this output file peerMagic_HZ.txt (https://github.com/altsheets/chaincountdown/blob/master/tools/peerMagic_HZ.txt).

I still did not really know how to speak to peernodeservers. Is the port 7774 communication really not documented other than so (https://bitcointalk.org/index.php?topic=823785.msg13661471#msg13661471)? During my first attempt,  I covered only ~2/3 of the network:

BUT that way, I could NOT find which is api.nhzcrypto.org (http://api.nhzcrypto.org:7776).
So ... something in my strategy is not perfect yet. I might not be finding all nodes!
Happy about any suggestion.

MaWo's script (https://bitcointalk.org/index.php?topic=823785.msg13656332#msg13656332) helped ... and after a lot of fiddling, I found the way. Yiehah. (https://bitcointalk.org/index.php?topic=823785.msg13661471#post_solved)

Now ChainCountDown is more resilient, because instead of 1 server per coin, I hardcoded 6 and 7 addresses (https://github.com/altsheets/chaincountdown/blob/09f23c778f65036607a1cddc0f1ffaff8d5a4b1c/config.py#L23-L46) with open API. That should do the trick.


If you like this, show it:
  NHZ-Q675-SGBG-LQ43-D38L6   or   NXT-CMKU-ZQYK-V6CD-9UHF4
  or hard cash [BTC] 13whfNFT2dse7eiuNXVkuJivjxqSXHrW4j
EDIT: retweet (https://twitter.com/altsheets/status/691041143396655104)

EDITED heavily, to avoid repetition below. See this post (https://nxtforum.org/index.php?topic=9735.msg207212#msg207212) for the continuation of this.
Title: Re: chaincountdown - blockchain aware countdown timer with image server
Post by: altsheets on January 25, 2016, 06:32:27 pm
chaincountdown v18

HZ peer network - analysis

Once the above crawler was ready, a natural next step was to make a network graph from the collected data.

This is how it looks:


Two nodes (green spheres) are connected with edges (gray lines) if they call each other 'peers'.
The more peers a node has (that is called the degree of a node), the bigger its sphere.

Strange shape, isn't it?

The network is very dense, so most nodes end up drawn into one big ball.
But those which are not, were catching my attention. Irregularities are interesting.

I tried an automatic clustering algorithm (i.e. "color nodes equal which are similar"),
called Louvain communities, and those nodes on the top right are clearly recognized
to be different from the all other ones:

(https://raw.githubusercontent.com/altsheets/chaincountdown/master/tools/networkanalysis/HZ_20160125-105105_network-20160125-112807_a_Louvain.png) (https://raw.githubusercontent.com/altsheets/chaincountdown/master/tools/networkanalysis/HZ_20160125-105105_network-20160125-112807_a_Louvain.png)

You also see some quantitative measurements of network properties. For example:
"Path length" means how many steps a transaction needs from one node to another one.
"Diameter=4" says, it takes maximum 4 steps for a transaction to reach any other node,
and "characteristic path length" that on average only 1.769 steps.

The "Number of shared neighbors" is interesting. Many nodes are almost copies to each other.

But back to the question ... what is making those nodes at the top right so
different, that they are drawn so far out of the main bulk? Ah ... idea:
I checked for the HZ version ("NHZ V5.4") of each node, and colored them:

(https://raw.githubusercontent.com/altsheets/chaincountdown/master/tools/networkanalysis/HZ_20160125-105105_network-20160125-112807_b_version-numbers.png) (https://raw.githubusercontent.com/altsheets/chaincountdown/master/tools/networkanalysis/HZ_20160125-105105_network-20160125-112807_b_version-numbers.png)

And now it was all clear: The nodes on the top right ... should be -but have not been- updated :-)
The mainstream (yellow) nodes do not like nostalgic, old-fashioned nodes - and do not connect to them.

The three V5 versions are mixing well, the network graph is showing that too.

Riddle solved.

By the way, here is the first picture with which I realized that I had probably hit the right reason:

(https://raw.githubusercontent.com/altsheets/chaincountdown/master/tools/networkanalysis/HZ_20160125-105105_network-20160125-112807_versions.clu_colored_KK-inside-clusters.png) (https://raw.githubusercontent.com/altsheets/chaincountdown/master/tools/networkanalysis/HZ_20160125-105105_network-20160125-112807_versions.clu_colored_KK-inside-clusters.png)

The placement of the nodes is a bit artificial, it uses a "optimize inside clusters only" layouting.
But it gives immediate feedback that coloring ("NHZ versions") has something to do with graph structure.
The "NHZ 4.0e" version still has a lot of fans  ;) (RED nodes).

... Anyways, back on track:

Now I dropped all older-than-V5 nodes from the picture - and 160 IP addresses remained:

(https://raw.githubusercontent.com/altsheets/chaincountdown/master/tools/networkanalysis/HZ_20160125-105105_network-20160125-112807_c_version5only.png) (https://raw.githubusercontent.com/altsheets/chaincountdown/master/tools/networkanalysis/HZ_20160125-105105_network-20160125-112807_c_version5only.png)

As they are hyperconnected, with many more links between them than needed
(diameter 2, average path length 1.722, average peers 44.225 / 159 = 28% of all nodes)
it is very difficult to get a meaningful automatic layout.

What I employed is the automatic partitioning "BlockModelling" by "Structural Equivalence"
... which identified 7 nodes (blue) to be different from all the other >150 ones.
They also seem to be the ones with the largest degree. I placed them manually.

At the bottom right you see a (log-log) histogram of degree versus number-of-such-nodes.

That's it, for now.

A typical next question of network analysis would be "resilience":
* ... against random failure: How many nodes have to fail until the network is fragmenting.
* ... against attacks: How many of the most connected nodes have to fail, until the network is fragmenting.

That can be examined in numerical simulations. Perhaps you want to try, then all network files are here (https://github.com/altsheets/chaincountdown/tree/master/tools/networkanalysis/networks-and-partitions) ...

That was a really enjoyable, geeky weekend. What had started as a short & simple
user question "your image server is down" ... evolved into a whole research project.

Fascinating, no?


If that was entertaining, enlightening, cool, or simply ... interesting
- then please now consider a way to give back (http://altsheets.ddns.net/give/). Thanks. You're good.