Nxt Forum

Nxt Discussion => Nxt Software Releases => Official Nxt Releases => Topic started by: Jean-Luc on April 19, 2015, 08:46:27 pm

Title: NRS v1.5.3e
Post by: Jean-Luc on April 19, 2015, 08:46:27 pm
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Release 1.5.3e

https://bitbucket.org/JeanLucPicard/nxt/downloads/nxt-client-1.5.3e.zip

sha256:

393a71bc869b7dafb2830eef2d29665f24c7c1c64741f3ecc103a088c4f85ac3  nxt-client-1.5.3e.zip


This is a development release for testing only. Source code is not provided.


Change log:

This is an experimental release. It is a required update for all testnet nodes,
but is also possible to run on main net.

This update adds the Prunable Tagged Data feature, planned to be enabled at the
voting system block too.


Prunable tagged data are similar to prunable plain messages with no recipient,
but with additional searchable metadata fields added. This feature can be used
for decentralized and trustless distribution of small (up to 42k, including
the metadata) pieces of data, which are by default stored for two weeks only
(24h on testnet), but can optionally be stored longer or indefinitely by some
nodes, and can be verified against the blockchain even after their expiration.

Currently each tagged data can have the following fields, in addition to the
data itself: name (required), description, tags, type, isText, filename.

Name, description and tags are indexed and searchable using Lucene. All data
and metadata is prunable, after pruning only a single 32 byte hash remains.

Fee for either uploading or extending tagged data is based on the total data
size (including metadata), and is 1 NXT for up to 1k bytes, 0.1 NXT for each
1k above, up to the limit of 42k.


New APIs:

UploadTaggedData - create and broadcast new tagged data.

ExtendTaggedData - extend the expiration time of already uploaded tagged data.
If the data is still available, only the transaction id is needed. If not,
already pruned data can be resurrected by including (in addition to the
original transaction id) all of its fields too, i.e. name, description, etc.
Anyone can submit an extension, not only the original uploader. Each extend
transaction increases the expiration deadline by two weeks (24 h on testnet).
Extending an existing tagged data from another account does not change the
original submitter account id by which it is indexed and searchable.

VerifyTaggedData - used to verify expired tagged data downloaded from another
node, against the hash in the current blockchain.

SearchTaggedData - full text search on name, tags, and description, optionally
filtered by submitter account.

GetTaggedData - retrieve tagged data by transaction id.

GetAccountTaggedData - retrieve all tagged data submitted by a given account.

GetAllTaggedData - retrieve all existing tagged data.

GetDataTags - returns the distinct tags of all tagged data, with the number
of data having each tag.

GetDataTagsLike - prefix search of data tags.

GetDataTagCount - the total number of distinct tags.

All the Get* and Search* APIs above can only retrieve tagged data that has
not been pruned yet.

There is no client UI support for any of these APIs. It is expected that
specialized third party plugins will be developed for specific tagged data
use cases, depending on file type and target user group.

Currently the pruning of tagged data is controlled by the same configuration
properties that are used for prunable messages expiration.

Same as with prunable messages, when using broadcastTransaction API to submit
an already signed tagged data upload transaction, the prunable parts must be
in the prunableAttachmentJSON parameter in case transactionBytes parameter is
used. If submitting transactionJSON, it already has the prunable parts.


New debug APIs:

GetAllBroadcastedTransactions - if transaction rebroadcasting is enabled,
returns the unconfirmed transactions that have been broadcasted from this node
but have not yet been received back from a peer.

GetAllWaitingTransactions - returns the unconfirmed transactions that are
currently temporarily kept in memory during transaction processing.

RebroadcastUnconfirmedTransactions - if transaction rebroadcasting is enabled,
causes transactions currently in the unconfirmed pool to be broadcasted to
peers again, until received back.


Other changes and bugfixes:

Improved handling of invalid transactions during block generation. Fixed
validation of AE order cancellation transactions. Fixed a memory leak in
processing of unconfirmed transactions. Set the default limit on waiting
unconfirmed transactions nxt.maxUnconfirmedTransactions to 1000.

The DGS goods delivery data size will be limited to 1000 bytes after VS block.

Bugfixes in prunable messages processing.

Fixed phasing with whitelisted approval accounts.

On testnet, this release will perform a full rescan on first start with
rollback to height 263895.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iQIcBAEBAgAGBQJVNA9YAAoJEFOhyXc7+e2AH/oQAO8GUyGVaqC/64pdO/1AM4Hx
ID0gbtGxPT8hTpifnS9QpKJKo8/YeDtyh4OmGn9pCnjYjO9wl3TyjIqi9sxQOmYX
mjo9p/C/NK2y2JqwAKIfwOTOEqG7I5iseE+kEH6lDFA50jHONJcj7q+UYM0LrfEr
CWZTQkf0jVNCoYqY+oZ5aSOhMS2BPl7tjhX6v3Y7xBgtq5NT/OALJykC3gkcDZ7A
w5B+9Ip/VUwtX2jVStunScQc6vrfT52seUBi8XzduRQAPoa7mNB5OnlTmE52glW1
uXwDFOD1OJgWFHNwDOmPAqmPf+uvXOv4Nir+yKlMlMe/ntRUhV1TenqM2fMSs+bh
YG0CB+0VmPP/VgIERff3CLILXlXr9CpDCFI1wS4BgcNb+BbvrDoJwA+W0ebhcBbG
zIuki3y6HJQ0krTUiURGKrTHMKvuBL9rBq9pbqAfqbTkT6oHtZxj/in+PMycJ1V4
+DDE+qmb8BZuOjsV7iyRwqZpoxnOat/t0Ryri2ByKpgsmMLu91jivi+h59zXOW+i
5ZLdMK8yIuRJEOYKDEOub/pX4PV7R3OkMLAeerFBfZ5ekmOIJpNB4Qc5bBgz+ZOq
MTGM7AzHZ3gISOXsG5JQOorut0ARW9fhW1PDNk3/pmhQMXu6UgYx9Cl1zbglz9S6
Jg/uehgLcHbZ8uHj6FeH
=oJRe
-----END PGP SIGNATURE-----
Title: Re: NRS v1.5.3e
Post by: Sebastien256 on April 19, 2015, 08:57:05 pm
ho great! Surely there are some robots in Nxt devs team. haha :D
Title: Re: NRS v1.5.3e
Post by: yassin54 on April 19, 2015, 09:22:23 pm
Thank you!  ;)
Title: Re: NRS v1.5.3e
Post by: EvilDave on April 19, 2015, 11:23:14 pm
Thanks, J-L, both my TestNet nodes are grinding away on the rescan....
Title: Re: NRS v1.5.3e
Post by: qq2536007339 on April 20, 2015, 01:41:48 am
That was fast.
Title: Re: NRS v1.5.3e
Post by: NxtSwe on April 20, 2015, 06:04:31 am
My god, you are a machine J-L.
Keep it up! :)
Title: Re: NRS v1.5.3e
Post by: jabo38 on April 20, 2015, 06:17:39 am
Hahahaha, the other day Dave posted about 1.5.2 so I wanted to try it out.  I got it all synced and now there is a new version already.  Does JL sleep?  ;)   

Seriously, JL doesn't mess around.
Title: Re: NRS v1.5.3e
Post by: Riker on April 20, 2015, 07:53:31 am
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Release 1.5.3e

https://bitbucket.org/JeanLucPicard/nxt/downloads/nxt-client-setup-1.5.3e.exe

sha256:

6ff25a57a3a0692cd80d47a6011d86ed4a4ff1813af353e362536bd8d036e8dd *nxt-client-setup-1.5.3e.exe

https://bitbucket.org/JeanLucPicard/nxt/downloads/nxt-client-setup-1.5.3e.jar

sha256:

7054cf7fbfbbd62afc98575dea0ac49773b9d97bd76fbc9b847ea389d0d790e4 *nxt-client-setup-1.5.3e.jar

This is an experimental release of the new integrated installer.
See: https://bitbucket.org/JeanLucPicard/nxt/issue/283/integrated-installer

The Jar file installation is compatible with Windows, Linux and Mac. To invoke it, open a command prompt, make sure Java 8 JRE is in your path then execute the following command:
java -jar <installer.name>.jar
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (MingW32)

iQEcBAEBAgAGBQJVNJR6AAoJECdelKfYMRZRG4EIAMQRjgZgmd+dmBZikmPLZSeQ
U/Yr2kcci+uNDDaCsjLmXH1lcRgop/HSKloKac5KGi9S1xA83qHEpyINI0f28NUX
tJ52csW9Pc4qqwMxgnlC/9PLkiST6jCHLEguh7ct3GHLKRED1PQX/w6yFpbGXTbK
BLYfMK/X0ERpQVjPM9SLY0S0NlCx+x8gEXJIkWH3olOkWMjPzIBmK7TmCbnJdUG8
zty4759MG0XS4xLqwd5drC1RDhzDOpkNMwSdrG+8+a64AK9UlJJtYa+2x1sUfq67
Ypx3T9LXwxagIzbPnsFlupDOSdoGKKRey3WDZW+PU90H59rPRT5KfvDaX7jv798=
=D1mp
-----END PGP SIGNATURE-----
Title: Re: NRS v1.5.3e
Post by: Tosch110 on April 20, 2015, 03:12:04 pm
Very nice those new APIs :)
Title: Re: NRS v1.5.3e
Post by: HCLivess on April 20, 2015, 07:42:19 pm
Hi, asset transfer does not work for me, says "feature not available"
Title: Re: NRS v1.5.3e
Post by: galeki on April 20, 2015, 08:56:43 pm
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Release 1.5.3e

https://bitbucket.org/JeanLucPicard/nxt/downloads/nxt-client-setup-1.5.3e.exe

sha256:

6ff25a57a3a0692cd80d47a6011d86ed4a4ff1813af353e362536bd8d036e8dd *nxt-client-setup-1.5.3e.exe

https://bitbucket.org/JeanLucPicard/nxt/downloads/nxt-client-setup-1.5.3e.jar

sha256:

7054cf7fbfbbd62afc98575dea0ac49773b9d97bd76fbc9b847ea389d0d790e4 *nxt-client-setup-1.5.3e.jar

This is an experimental release of the new integrated installer.
See: https://bitbucket.org/JeanLucPicard/nxt/issue/283/integrated-installer

The Jar file installation is compatible with Windows, Linux and Mac. To invoke it, open a command prompt, make sure Java 8 JRE is in your path then execute the following command:
java -jar <installer.name>.jar
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (MingW32)

iQEcBAEBAgAGBQJVNJR6AAoJECdelKfYMRZRG4EIAMQRjgZgmd+dmBZikmPLZSeQ
U/Yr2kcci+uNDDaCsjLmXH1lcRgop/HSKloKac5KGi9S1xA83qHEpyINI0f28NUX
tJ52csW9Pc4qqwMxgnlC/9PLkiST6jCHLEguh7ct3GHLKRED1PQX/w6yFpbGXTbK
BLYfMK/X0ERpQVjPM9SLY0S0NlCx+x8gEXJIkWH3olOkWMjPzIBmK7TmCbnJdUG8
zty4759MG0XS4xLqwd5drC1RDhzDOpkNMwSdrG+8+a64AK9UlJJtYa+2x1sUfq67
Ypx3T9LXwxagIzbPnsFlupDOSdoGKKRey3WDZW+PU90H59rPRT5KfvDaX7jv798=
=D1mp
-----END PGP SIGNATURE-----


Testing under windows8.1, the Java packaged installer and the tray icon are so cool, working smoothly.  ;)
Title: Re: NRS v1.5.3e
Post by: Jean-Luc on April 21, 2015, 09:36:07 am
Hi, asset transfer does not work for me, says "feature not available"
If this is on main net, and trying to add a message too, it is a known bug - messages default to prunable in 1.5, but this feature is not yet enabled on main net, and the checkbox to disable prunable is missing from most transaction send dialogs.
Title: Re: NRS v1.5.3e
Post by: Jean-Luc on April 21, 2015, 09:42:20 am
Very nice those new APIs :)
Should we set a bounty for the first torrent tracker plugin using the tagged data feature? It would be controversial to have in the default client, but is a legitimate use case for someone to provide via a plugin.

The tagged data APIs really need more testing, I only spent a week implementing this feature, there may be bugs.

Title: Re: NRS v1.5.3e
Post by: verymuchso on April 21, 2015, 09:47:50 am
and the checkbox to disable prunable is missing from most transaction send dialogs.

Then why not make a single dynamic transaction dialog instead of repeating the same thing over and over again.

Should we set a bounty for the first torrent tracker plugin using the tagged data feature? It would be controversial to have in the default client, but is a legitimate use case for someone to provide via a plugin.
The tagged data APIs really need more testing, I only spent a week implementing this feature, there may be bugs.

What do the tagged data API's do?

Guess I missed the OP.
Title: Re: NRS v1.5.3e
Post by: verymuchso on April 21, 2015, 10:04:52 am
About the pruning on timestamp..

Has it been considered to prune based on the total size of the stored prunable data instead of on it's timestamp?

It seems that if you would use a fixed size in bytes (set in a users config) you would be able to allow the bigger stronger (public) nodes to raise the default level simply by changing the max-prunable-total-size to for instance 10GB or more even. While normal users would have this set at 100MB or maybe even zero.

This instantly solves the "special" backup nodes thing which will simply be a change to the size.

Another thing that might be interesting is if you would allow the backup nodes to only backup certain messages, which of course need to be identified some way. This would allow technology providers building on NXT to host big backup nodes that only backup "their" messages (messaging apps come to mind here).
Title: Re: NRS v1.5.3e
Post by: verymuchso on April 21, 2015, 10:56:47 am
Another thing that might be interesting is if you would allow the backup nodes to only backup certain messages, which of course need to be identified some way. This would allow technology providers building on NXT to host big backup nodes that only backup "their" messages (messaging apps come to mind here).

I can imagine to use this for (almost) zero fee transactions where a low fee of 1 NXT is paid for a prunnable message of max 42KB, normal users would only store the hash but dedicated nodes could store the payload when they recognize it as belonging to them. (Jetcoin comes to mind)

1. Communities or businesses could that way support the network.
2. Have all the rewards of having free updates to the core software.
3. Get all the crypto magic that comes with the package
4. Easy no-config cluster setup (simply start a couple of nodes)
5. Plus huge amounts of storage of their data (almost) free of charge.

and the list goes on..

Of course you'd have to look at the effect of the low fee in combination with the message size and having the risk of being ddos'ed.

feedback?
Title: Re: NRS v1.5.3e
Post by: kushti on April 21, 2015, 03:15:51 pm
It seems 1.5.3e node is stuck @ 260822 . Rollback (by few K blocks) doesn't help.
Title: Re: NRS v1.5.3e
Post by: allwelder on April 21, 2015, 03:22:37 pm
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Release 1.5.3e

https://bitbucket.org/JeanLucPicard/nxt/downloads/nxt-client-setup-1.5.3e.exe

This means there will be on click install .exe client from1.5.x ?
Great.
Title: Re: NRS v1.5.3e
Post by: ScripterRon on April 21, 2015, 04:17:51 pm
It seems 1.5.3e node is stuck @ 260822 . Rollback (by few K blocks) doesn't help.
My testnet node (scripterron.dyndns.biz:6874) is at 265777.  It sounds like you might be stuck on a fork.  The block at 260822 is 11708484282988747400 on my node.  You can try deleting nxt_test_db and downloading from my node is you want to give it a try.
Title: Re: NRS v1.5.3e
Post by: abctc on April 21, 2015, 05:26:42 pm
It seems 1.5.3e node is stuck @ 260822 . Rollback (by few K blocks) doesn't help.
My testnet node (scripterron.dyndns.biz:6874) is at 265777.  It sounds like you might be stuck on a fork.  The block at 260822 is 11708484282988747400 on my node.  You can try deleting nxt_test_db and downloading from my node is you want to give it a try.
- my 1.5.3e testnode is OK too (265821).
Title: Re: NRS v1.5.3e
Post by: Bernard Lerring on April 21, 2015, 05:42:43 pm
Just trying out the 1.5x version for the first time. Been using 1.4x for a few months and I usually copy over "nxt_db" directory in Ubuntu to instantly have an up to date block chain.

Copying a 1.4x "nxt_db" directory over to a 1.5x installation doesn't work. Is the block chain stored in a different format?

I've stopped the client, deleted "nxt_db" and I'm letting it download naturally. I'm just curious.
Title: Re: NRS v1.5.3e
Post by: abctc on April 21, 2015, 05:59:27 pm
Copying a 1.4x "nxt_db" directory over to a 1.5x installation doesn't work.
- you are wrong. It works perfectly, as always (nxt_db - on mainnet).
Title: Re: NRS v1.5.3e
Post by: Bernard Lerring on April 21, 2015, 06:05:05 pm
Really? Maybe it copied wrong. I'll try again.

Edit: Nope. It starts eating resources and I get this:

Code: [Select]
2015-04-21 19:06:02 FINE: Database connection pool current size: 2
2015-04-21 19:06:02 FINE: Will apply sql:
DROP TABLE IF EXISTS poll
2015-04-21 19:06:02 FINE: Will apply sql:
DROP TABLE IF EXISTS vote
2015-04-21 19:06:02 FINE: Will apply sql:
CREATE TABLE IF NOT EXISTS vote (db_id IDENTITY, id BIGINT NOT NULL, poll_id BIGINT NOT NULL, voter_id BIGINT NOT NULL, vote_bytes VARBINARY NOT NULL, height INT NOT NULL)
2015-04-21 19:06:02 FINE: Will apply sql:
CREATE UNIQUE INDEX IF NOT EXISTS vote_id_idx ON vote (id)
2015-04-21 19:06:02 FINE: Will apply sql:
CREATE UNIQUE INDEX IF NOT EXISTS vote_poll_id_idx ON vote (poll_id, voter_id)
2015-04-21 19:06:02 FINE: Will apply sql:
CREATE TABLE IF NOT EXISTS poll (db_id IDENTITY, id BIGINT NOT NULL, account_id BIGINT NOT NULL, name VARCHAR NOT NULL, description VARCHAR, options ARRAY NOT NULL, min_num_options TINYINT, max_num_options TINYINT, min_range_value TINYINT, max_range_value TINYINT, finish_height INT NOT NULL, voting_model TINYINT NOT NULL, min_balance BIGINT, min_balance_model TINYINT, holding_id BIGINT, height INT NOT NULL)
2015-04-21 19:06:02 FINE: Will apply sql:
CREATE TABLE IF NOT EXISTS poll_result (db_id IDENTITY, poll_id BIGINT NOT NULL, result BIGINT, weight BIGINT NOT NULL, height INT NOT NULL)
2015-04-21 19:06:02 FINE: Will apply sql:
ALTER TABLE transaction ADD COLUMN IF NOT EXISTS phased BOOLEAN NOT NULL DEFAULT FALSE
Title: Re: NRS v1.5.3e
Post by: martismartis on April 21, 2015, 06:14:17 pm
Really? Maybe it copied wrong. I'll try again.

Edit: Nope. It starts eating resources and I get this:

Code: [Select]
2015-04-21 19:06:02 FINE: Database connection pool current size: 2
2015-04-21 19:06:02 FINE: Will apply sql:
DROP TABLE IF EXISTS poll
2015-04-21 19:06:02 FINE: Will apply sql:
DROP TABLE IF EXISTS vote
2015-04-21 19:06:02 FINE: Will apply sql:
CREATE TABLE IF NOT EXISTS vote (db_id IDENTITY, id BIGINT NOT NULL, poll_id BIGINT NOT NULL, voter_id BIGINT NOT NULL, vote_bytes VARBINARY NOT NULL, height INT NOT NULL)
2015-04-21 19:06:02 FINE: Will apply sql:
CREATE UNIQUE INDEX IF NOT EXISTS vote_id_idx ON vote (id)
2015-04-21 19:06:02 FINE: Will apply sql:
CREATE UNIQUE INDEX IF NOT EXISTS vote_poll_id_idx ON vote (poll_id, voter_id)
2015-04-21 19:06:02 FINE: Will apply sql:
CREATE TABLE IF NOT EXISTS poll (db_id IDENTITY, id BIGINT NOT NULL, account_id BIGINT NOT NULL, name VARCHAR NOT NULL, description VARCHAR, options ARRAY NOT NULL, min_num_options TINYINT, max_num_options TINYINT, min_range_value TINYINT, max_range_value TINYINT, finish_height INT NOT NULL, voting_model TINYINT NOT NULL, min_balance BIGINT, min_balance_model TINYINT, holding_id BIGINT, height INT NOT NULL)
2015-04-21 19:06:02 FINE: Will apply sql:
CREATE TABLE IF NOT EXISTS poll_result (db_id IDENTITY, poll_id BIGINT NOT NULL, result BIGINT, weight BIGINT NOT NULL, height INT NOT NULL)
2015-04-21 19:06:02 FINE: Will apply sql:
ALTER TABLE transaction ADD COLUMN IF NOT EXISTS phased BOOLEAN NOT NULL DEFAULT FALSE

Updating from 1.4 to 1.5 needs updating of database with new fields in it. It takes some time, be patient. For me it took about 1 hour.
Title: Re: NRS v1.5.3e
Post by: Bernard Lerring on April 21, 2015, 07:28:22 pm
I thought it was maybe a new way of indexing or interpolating parts of the block chain. So, once a chain is incorporated into a 1.5x client it is different from a 1.4x chain?
Title: Re: NRS v1.5.3e
Post by: toenu on April 21, 2015, 07:33:27 pm
Just tried out the tagged data API.

What is the "type" field intended for? It seems to take arbitrary data but is not searchable.
Is it meant for file extensions/mime types, or more a general description like "video" or something else. (i.e. when using for torrents)

Also, I created data with the tags "test,asdf,1234". For some reason it takes "test" as one tag and "asdf,1234" as another.
Title: Re: NRS v1.5.3e
Post by: martismartis on April 21, 2015, 07:36:51 pm
I thought it was maybe a new way of indexing or interpolating parts of the block chain. So, once a chain is incorporated into a 1.5x client it is different from a 1.4x chain?

Let's name it rescan and optimization of database :)
Title: Re: NRS v1.5.3e
Post by: Bernard Lerring on April 21, 2015, 07:56:18 pm
Nice. So I take it that multiple assets/asset value etc. will load up much faster on the optimized block chain? It sometimes takes about 30 seconds on my old netbook in 1.4x. Still fast on my normal PC though.
Title: Re: NRS v1.5.3e
Post by: neofelis on April 21, 2015, 09:35:06 pm
How do I start forging?  The "forging" link in the upper left is gone.

NVMD - figured it out. ::)
Title: Re: NRS v1.5.3e
Post by: jabo38 on April 21, 2015, 10:33:47 pm
probably the wrong place to ask, but I looked and couldn't find one.  Could somebody please send me some test NXT please? NXT-2SBM-BM59-T7XC-8TEFU
Title: Re: NRS v1.5.3e
Post by: EvilDave on April 22, 2015, 12:15:04 am
@Jabo......
https://nxtforum.org/testnet/some-testnxt-to-test-asset-exchange/msg174890/#msg174890

Sending you a 1000 TestNXT, anyway, as i have more than enough for my needs. ;D
Title: Re: NRS v1.5.3e
Post by: jabo38 on April 22, 2015, 05:21:57 am
Thanks!  I'll bookmark that. 
Title: Re: NRS v1.5.3e
Post by: Jean-Luc on April 22, 2015, 10:32:05 am
What is the "type" field intended for? It seems to take arbitrary data but is not searchable.
Is it meant for file extensions/mime types, or more a general description like "video" or something else. (i.e. when using for torrents)
I haven't decided yet actually. I intended it to be a mime type, to be used to tell the browser what to do when receiving such file. But we can also leave it free from, and if not a valid mime type default to serving it as a binary file or plain text, if isText=true, and then specific plugins can use it for other purposes.

It is not searchable by the full text search, but I can add it as an optional filter to all APIs. We could also add a separate field, "channel", if we want to group data by some more general type, e.g. news, torrents, leaks, pics, and leave the type field strictly for mime type. As long as the basic framework of uploading, searching, extending, and pruning the data is working, defining what exactly metadata fields they should have is not difficult, we just need to agree on what would be most useful.

Quote
Also, I created data with the tags "test,asdf,1234". For some reason it takes "test" as one tag and "asdf,1234" as another.
Strange indeed, but this is how the Lucene analyzer does it. It has some heuristics whether to split at punctuation or not, I believe trying to detect product numbers, hostnames, etc. To be safe, use whitespace as separator. I prefer not to mess with this, because the text processing needs to be consistent both at indexing and at searching time, and we don't have any control over the defaults used by H2 (which is also using an old Lucene version). Later we may handle search independently of H2, creating our own more customizable Lucene index.
Title: Re: NRS v1.5.3e
Post by: box1413 on April 22, 2015, 03:25:19 pm
im curious to why the max message size is 42kb? since its going to be pruned anyway, why not set a higher limit? the pruning still it doesnt stop someone to concating multiple messages to form a bigger file.
Title: Re: NRS v1.5.3e
Post by: crimi on April 22, 2015, 05:52:04 pm
im curious to why the max message size is 42kb? since its going to be pruned anyway, why not set a higher limit? the pruning still it doesnt stop someone to concating multiple messages to form a bigger file.

I guess total blocksize, would be at somepoint hard to target/validate the 1 min blocktime anymore.
Title: Re: NRS v1.5.3e
Post by: Jean-Luc on April 22, 2015, 09:55:43 pm
The total block payload size is still around 44k (which in practice can mean more than 220k when expressed as json). Increasing this risks creating problems with fork resolution, because a node may need to download multiple blocks and switch between forks before the time for the next block comes. If the transaction payload size is to be increased further, this has to be done with downloading the prunable data out of band, after the fork resolution, but this is too complicated to be done in this release.
Title: Re: NRS v1.5.3e
Post by: jl777 on April 23, 2015, 12:12:34 am
The total block payload size is still around 44k (which in practice can mean more than 220k when expressed as json). Increasing this risks creating problems with fork resolution, because a node may need to download multiple blocks and switch between forks before the time for the next block comes. If the transaction payload size is to be increased further, this has to be done with downloading the prunable data out of band, after the fork resolution, but this is too complicated to be done in this release.
Can we have 10MB blocks?

It would be very cool to just allow for this possibility with the next hard fork. As bitcoiners debate ad nauseum about this, NXT can just do it.

I doubt any block will fill up so it would just be a theoretical limit. It does concern me that a single bot can totally fill up the current block via InstantDEX arbitrages

James

P.S. We can even use 1MB (or 1.7MB) blocks as the equivalent of 10MB BTC blocks due to blocktime difference
Title: Re: NRS v1.5.3e
Post by: ScripterRon on April 23, 2015, 12:43:03 am
Can we have 10MB blocks?
This is something I ran into as I was implementing the new WebSocket support.  It turns out that NRS needs to be able to download up to 719 blocks at a time in order to handle fork resolution.  So the maximum response message size becomes the upper limit on the average block size unless the fork resolution code is redesigned.  Or as JL said, non-essential data needs to be downloaded separately to reduce the message size.
Title: Re: NRS v1.5.3e
Post by: jl777 on April 23, 2015, 01:06:44 am
Can we have 10MB blocks?
This is something I ran into as I was implementing the new WebSocket support.  It turns out that NRS needs to be able to download up to 719 blocks at a time in order to handle fork resolution.  So the maximum response message size becomes the upper limit on the average block size unless the fork resolution code is redesigned.  Or as JL said, non-essential data needs to be downloaded separately to reduce the message size.
How often does it need to get all 719 blocks? if it is rare maybe we can live with a theoretical slowdown. I am talking about real 1 MB blocks full of transactions. If we have to hard fork for this, might as well do it now.

James
Title: Re: NRS v1.5.3e
Post by: toenu on April 23, 2015, 06:14:01 am
What is the "type" field intended for? It seems to take arbitrary data but is not searchable.
Is it meant for file extensions/mime types, or more a general description like "video" or something else. (i.e. when using for torrents)
I haven't decided yet actually. I intended it to be a mime type, to be used to tell the browser what to do when receiving such file. But we can also leave it free from, and if not a valid mime type default to serving it as a binary file or plain text, if isText=true, and then specific plugins can use it for other purposes.

It is not searchable by the full text search, but I can add it as an optional filter to all APIs. We could also add a separate field, "channel", if we want to group data by some more general type, e.g. news, torrents, leaks, pics, and leave the type field strictly for mime type. As long as the basic framework of uploading, searching, extending, and pruning the data is working, defining what exactly metadata fields they should have is not difficult, we just need to agree on what would be most useful.

Files could be stored as a Data-URI (https://en.wikipedia.org/wiki/Data_URI_scheme), which already contains the mime type. (e.g. "data:application/x-bittorrent;base64,ZDg6YW5ub3VuY2U....")
This is very practical as it can just be opened in the browser. In that case type could be omitted or used for something else.

For categorization of content in addition to tags it would definitely be useful to have type / channel as a search attribute.
elective-stereophonic
elective-stereophonic
assembly
assembly