Topic: NRS v1.5.9 (Read 50364 times)

Riker · « **Reply #160 on:** June 01, 2015, 04:57:09 pm »

You can open a command line window then invoke run.bat from there to get a better error message.
Most likely, you need to install Java 8.

th4o · « **Reply #161 on:** June 01, 2015, 05:37:39 pm »

Quote from: Tosch110 on May 30, 2015, 10:19:58 am

Quote from: ScripterRon on May 30, 2015, 03:38:19 am
Quote from: Tosch110 on May 30, 2015, 01:41:44 am
I am posting this because the script told me to

Code: [Select]
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: unable to create new native thread ... Caused by: java.lang.OutOfMemoryError: unable to create new native thread [....] Caused by: org.h2.jdbc.JdbcSQLException: Database may be already in use: "Locked by another process". java.lang.OutOfMemoryError: unable to create new native thread

"Unable to create a new native thread" can be caused by the heap being too small or by using a 32-bit JVM on a 64-bit system. What is your heap size (-Xmx option)? If you are on a 64-bit system, what is displayed when you enter 'java -version'?

"Database already open" can occur if NRS is started while a previous instance is still running.

Quote
java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

I have a 1&1 server and running Nxt is really problematic, it keeps crashing regularly. (I have to check every 1-2 days to be sure, you never know when it crashes next time. Sometimes it runs for a week or two, it has been running for some weeks without problems. But most of the time for the last months, it crashed within 2-3 weeks or some days after starting it)

These are the commands I am using, varying also to get some experience which works best:

Quote
nohup java -cp nxt.jar:lib/*:conf nxt.Nxt &
nohup java -Xms300m -Xmx330m -cp nxt.jar:lib/*:conf nxt.Nxt &
nohup java -Xmx1024M -cp nxt.jar:lib/*:conf nxt.Nxt &

The system information look like it should run without any problems:

I´ve the same issue.

NRS keeps crashing with this logged:
2015-05-31 16:26:09 SEVERE: CRITICAL ERROR. PLEASE REPORT TO THE DEVELOPERS.
java.lang.OutOfMemoryError: unable to create new native thread

java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

NRS has been running fine since month on this VPS

Tosch110 · « **Reply #162 on:** June 02, 2015, 07:03:27 pm »

Quote from: th4o on June 01, 2015, 05:37:39 pm

Quote from: Tosch110 on May 30, 2015, 10:19:58 am
Quote from: ScripterRon on May 30, 2015, 03:38:19 am
Quote from: Tosch110 on May 30, 2015, 01:41:44 am
I am posting this because the script told me to

Code: [Select]
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: unable to create new native thread ... Caused by: java.lang.OutOfMemoryError: unable to create new native thread [....] Caused by: org.h2.jdbc.JdbcSQLException: Database may be already in use: "Locked by another process". java.lang.OutOfMemoryError: unable to create new native thread

"Unable to create a new native thread" can be caused by the heap being too small or by using a 32-bit JVM on a 64-bit system. What is your heap size (-Xmx option)? If you are on a 64-bit system, what is displayed when you enter 'java -version'?

"Database already open" can occur if NRS is started while a previous instance is still running.

Quote
java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

I have a 1&1 server and running Nxt is really problematic, it keeps crashing regularly. (I have to check every 1-2 days to be sure, you never know when it crashes next time. Sometimes it runs for a week or two, it has been running for some weeks without problems. But most of the time for the last months, it crashed within 2-3 weeks or some days after starting it)

These are the commands I am using, varying also to get some experience which works best:

Quote
nohup java -cp nxt.jar:lib/*:conf nxt.Nxt &
nohup java -Xms300m -Xmx330m -cp nxt.jar:lib/*:conf nxt.Nxt &
nohup java -Xmx1024M -cp nxt.jar:lib/*:conf nxt.Nxt &

The system information look like it should run without any problems:

I´ve the same issue.

NRS keeps crashing with this logged:
2015-05-31 16:26:09 SEVERE: CRITICAL ERROR. PLEASE REPORT TO THE DEVELOPERS.
java.lang.OutOfMemoryError: unable to create new native thread

java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

NRS has been running fine since month on this VPS

Ok, I have been experimenting a bit with my problem, maybe you got the same issue still.

Please correct me when I am wrong, this is just my interpretation of my problem:

Each time Nxt connects to a new peer, it creates a "native thread", which somehow increases the Nxt heap size and might exceed my memory limit. So for Nxt, I should at first reduce the memory usage with for example:

Code: [Select]

java -Xms300m -Xmx330m -cp nxt.jar:lib/*:conf nxt.Nxt
More on this here. How to fix "java.lang.OutOfMemoryError: unable to create new native thread"

Now, to limit the amount of threads Nxt creates, I can vary the following parameter in my config file:
(create a new nxt.properties file in nxt/conf and add the following lines. Start Nxt after editing.)

Code: [Select]

# Maximum number of inbound connections
nxt.maxNumberOfInboundConnections=10

# Maximum number of outbound connections
nxt.maxNumberOfOutboundConnections=8

#Maximum number of connected peers
nxt.maxNumberOfConnectedPublicPeers=8

In this example it is reduced to a very low limit. But I am experimenting with the options and it looks like this works so far for me. Maybe someone can give an advice how to best tune these?

davethetrousers · « **Reply #163 on:** June 02, 2015, 07:28:03 pm »

Quote from: Tosch110 on June 02, 2015, 07:03:27 pm

Code: [Select]
# Maximum number of inbound connections nxt.maxNumberOfInboundConnections=10 # Maximum number of outbound connections nxt.maxNumberOfOutboundConnections=8 #Maximum number of connected peers nxt.maxNumberOfConnectedPublicPeers=8
In this example it is reduced to a very low limit. But I am experimenting with the options and it looks like this works so far for me. Maybe someone can give an advice how to best tune these?

Inbound cnct's are for people pulling blocks from you, outbound is for you pulling from other people. For a private non-hallmark, inbound can be really small, your 10 are fine. A dedicated hallmark of course can make use of many tens or hundreds of these to supply nodes with blocks quickly.
Outbound should be raised somewhat, so that you're always up-to-date. For a full initial sync, you'll need at least 40-50 to have it run fast.

Public nodes I'm not completely sure about, I assume those are nodes with static address (an URL) and a hallmark. The "backbone network", if you will. Being connected to some of those is always beneficial. Raise those a bit as well, although somewhat less than the outbounds.

ScripterRon · « **Reply #164 on:** June 02, 2015, 07:32:49 pm »

Quote from: Tosch110 on June 02, 2015, 07:03:27 pm

Ok, I have been experimenting a bit with my problem, maybe you got the same issue still.

Please correct me when I am wrong, this is just my interpretation of my problem:

Each time Nxt connects to a new peer, it creates a "native thread", which somehow increases the Nxt heap size and might exceed my memory limit. So for Nxt, I should at first reduce the memory usage with for example:

Code: [Select]
java -Xms300m -Xmx330m -cp nxt.jar:lib/*:conf nxt.Nxt
More on this here. How to fix "java.lang.OutOfMemoryError: unable to create new native thread"

Now, to limit the amount of threads Nxt creates, I can vary the following parameter in my config file:
(create a new nxt.properties file in nxt/conf and add the following lines. Start Nxt after editing.)

Code: [Select]
# Maximum number of inbound connections nxt.maxNumberOfInboundConnections=10 # Maximum number of outbound connections nxt.maxNumberOfOutboundConnections=8 #Maximum number of connected peers nxt.maxNumberOfConnectedPublicPeers=8
In this example it is reduced to a very low limit. But I am experimenting with the options and it looks like this works so far for me. Maybe someone can give an advice how to best tune these?

There is not a thread per connection. Jetty creates threads based on the number of concurrent inbound network requests. But the more inbound connections you have, the greater the chance for concurrent I/O operations. So reducing the number of inbound connections will reduce the number of concurrent requests, although it is not a one-for-one relationship.

One factor driving the number of inbound connections is the need for a peer to find hallmarked nodes. The only way to do this at present is to keep connecting to random peers until you find enough hallmarked peers. Unfortunately, you are left with a lot of non-hallmarked peers (sometimes more than 100). NRS 1.5 introduced an upper limit on the number of outbound connections. This causes non-hallmarked peers to be disconnected during the search for hallmarked peers once this limit is reached. Hopefully, as more nodes switch to 1.5, the number of inbound connections experienced by each node in the network will decrease.

NRS will also use multiple threads when downloading blocks. This change will take effect at block 445000 when the hard fork occurs. So even if you solve your problem now, it might occur again when parallel downloads utilizing multiple threads becomes active.

Aside from threads, heap usage is likely to increase as the database size increases since H2 needs to build result sets for database query operations. There are H2 tuning parameters to limit the size of result sets but this just causes temporary files to be created to hold the result set, resulting in significantly slower performance due to disk I/O.

ScripterRon · « **Reply #165 on:** June 02, 2015, 07:47:05 pm »

Quote from: davethetrousers on June 02, 2015, 07:28:03 pm

Quote from: Tosch110 on June 02, 2015, 07:03:27 pm
Code: [Select]
# Maximum number of inbound connections nxt.maxNumberOfInboundConnections=10 # Maximum number of outbound connections nxt.maxNumberOfOutboundConnections=8 #Maximum number of connected peers nxt.maxNumberOfConnectedPublicPeers=8
In this example it is reduced to a very low limit. But I am experimenting with the options and it looks like this works so far for me. Maybe someone can give an advice how to best tune these?

Inbound cnct's are for people pulling blocks from you, outbound is for you pulling from other people. For a private non-hallmark, inbound can be really small, your 10 are fine. A dedicated hallmark of course can make use of many tens or hundreds of these to supply nodes with blocks quickly.
Outbound should be raised somewhat, so that you're always up-to-date. For a full initial sync, you'll need at least 40-50 to have it run fast.

Public nodes I'm not completely sure about, I assume those are nodes with static address (an URL) and a hallmark. The "backbone network", if you will. Being connected to some of those is always beneficial. Raise those a bit as well, although somewhat less than the outbounds.

It doesn't matter if your node is hallmarked. You are a public node if you announce your address. The hallmark comes into play when you need to get new blocks. Hallmarked nodes in your connected peer list have a higher chance of being selected based on the account balance. But NRS will also randomly pick a non-hallmarked node to get blocks from.

nxt.maxNumberOfConnectedPublicPeers determines the minimum number of outbound connections. If hallmark protection is not enabled, then NRS will stop as soon as it has this many connections. If hallmark protection is enabled, then NRS will continue creating outbound connections until it has connected to this number of hallmarked nodes. Note that non-hallmarked nodes remain connected unless the maximum number of outbound connections has been reached.

At the present time, having multiple outbound connections doesn't make it faster since all blocks are downloaded from a single peer (this is done sequentially in groups of 720 blocks so a different download peer might be selected for each group). Once parallel download is active, each group of 720 blocks will be downloaded in parallel from 20 peers in 36-block segments (if you have that many outbound connections). This will result in faster downloads as well as reducing the network load on each download peer. So having at least 20 outbound connections (hallmarked and non-hallmarked) is a good idea.

Tosch110 · « **Reply #166 on:** June 02, 2015, 11:21:00 pm »

Ok, let me try to sum it up:

I have the problem java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: unable to create new native thread

To solve this at least step by step is doing:

Decrease memory usage of Nxt with -Xmx and -Xms parameters while starting:

java -Xms300m -Xmx330m -cp nxt.jar:lib/*:conf nxt.Nxt

Before starting, I can modify my config to connect to less peers, which will decrease the RAM usage. Inserting into nxt/conf/nxt.properties (create if not exist)

# Maximum number of inbound connections
nxt.maxNumberOfInboundConnections=10

# Maximum number of outbound connections (increase for better blockchain data)
nxt.maxNumberOfOutboundConnections=20

#Maximum number of connected peers
nxt.maxNumberOfConnectedPublicPeers=20

So far fine?

Riker · « **Reply #167 on:** June 03, 2015, 09:30:57 am »

Quote from: Tosch110 on June 02, 2015, 11:21:00 pm

I have the problem java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: unable to create new native thread

Folks, the message above is related to running out of memory space for Java threads stack space not heap space.
This message can happen in one of the following cases:
1. You are running a 32 bit JRE with very large heap allocation (over 1GB possibly more) thus leaving only small space for thread stack. This is even more problematic on 64 bit Windows since thread stack allocation is larger on 64 bit. If you are running on a 64 bit Windows always use a 64 bit server jre.
2. From some reason our code opens too many threads, usually you need 1000's of threads to reproduce this message.

Next time you run into this message invoke the new GetStackTraces API. This will write a thread dump to the nxt.log which will show us if the NRS is creating too many threads from some reason.

coinomat · « **Reply #168 on:** June 03, 2015, 10:33:20 am »

Everything looks very neat, you guys are my heros.
NXT is the BEST!

Tosch110 · « **Reply #169 on:** June 03, 2015, 01:26:00 pm »

Quote from: lyaffe on June 03, 2015, 09:30:57 am

Quote from: Tosch110 on June 02, 2015, 11:21:00 pm
I have the problem java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: unable to create new native thread

Folks, the message above is related to running out of memory space for Java threads stack space not heap space.
This message can happen in one of the following cases:
1. You are running a 32 bit JRE with very large heap allocation (over 1GB possibly more) thus leaving only small space for thread stack. This is even more problematic on 64 bit Windows since thread stack allocation is larger on 64 bit. If you are running on a 64 bit Windows always use a 64 bit server jre.
2. From some reason our code opens too many threads, usually you need 1000's of threads to reproduce this message.

Next time you run into this message invoke the new GetStackTraces API. This will write a thread dump to the nxt.log which will show us if the NRS is creating too many threads from some reason.

Code: (http://localhost:7876/nxt?requestType=getStackTraces) [Select]

array(121) { [0]=> object(stdClass)#2 (4) { ["trace"]=> array(8) { [0]=> string(40) "java.lang.System.nanoTime(Native Method)" [1]=> string(118) "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2081)" [2]=> string(109) "java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)" [3]=> string(108) "java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)" [4]=> string(77) "java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)" [5]=> string(79) "java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)" [6]=> string(79) "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)" [7]=> string(37) "java.lang.Thread.run(Thread.java:745)" } ["name"]=> string(19) "Scheduler-927327686" ["id"]=> int(168) ["state"]=> string(8) "RUNNABLE" } [1]=> object(stdClass)#3 (4) { ["trace"]=> array(9) { [0]=> string(35) "sun.misc.Unsafe.park(Native Method)" [1]=> string(70) "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)" [2]=> string(118) "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)" [3]=> string(109) "java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)" [4]=> string(108) "java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)" [5]=> string(77) "java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)" [6]=> string(79) "java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)" [7]=> string(79) "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)" [8]=> string(37) "java.lang.Thread.run(Thread.java:745)" } ["name"]=> string(19) "Scheduler-584559462" ["id"]=> int(167) ["state"]=> string(13) "TIMED_WAITING" } [2]=> object(stdClass)#4 (4) { ["trace"]=> array(9) { [0]=> string(35) "sun.misc.Unsafe.park(Native Method)" [1]=> string(70) "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)"

[...]
[116]=> object(stdClass)#138 (4) { ["trace"]=> array(3) { [0]=> string(36) "java.lang.Object.wait(Native Method)" [1]=> string(52) "org.h2.store.WriterThread.run(WriterThread.java:103)" [2]=> string(37) "java.lang.Thread.run(Thread.java:745)" } ["name"]=> string(17) "H2 Log Writer NXT" ["id"]=> int(13) ["state"]=> string(13) "TIMED_WAITING" } [117]=> object(stdClass)#139 (4) { ["trace"]=> array(3) { [0]=> string(37) "java.lang.Thread.sleep(Native Method)" [1]=> string(44) "org.h2.store.FileLock.run(FileLock.java:517)" [2]=> string(37) "java.lang.Thread.run(Thread.java:745)" } ["name"]=> string(56) "H2 File Lock Watchdog /home/tosch/nxt/nxt_db/nxt.lock.db" ["id"]=> int(12) ["state"]=> string(13) "TIMED_WAITING" } [118]=> object(stdClass)#140 (4) { ["trace"]=> array(0) { } ["name"]=> string(17) "Signal Dispatcher" ["id"]=> int(4) ["state"]=> string(8) "RUNNABLE" } [119]=> object(stdClass)#141 (4) { ["trace"]=> array(4) { [0]=> string(36) "java.lang.Object.wait(Native Method)" [1]=> string(60) "java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)" [2]=> string(60) "java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)" [3]=> string(63) "java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)" } ["name"]=> string(9) "Finalizer" ["id"]=> int(3) ["state"]=> string(7) "WAITING" } [120]=> object(stdClass)#142 (4) { ["trace"]=> array(3) { [0]=> string(36) "java.lang.Object.wait(Native Method)" [1]=> string(38) "java.lang.Object.wait(Object.java:502)" [2]=> string(64) "java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)" } ["name"]=> string(17) "Reference Handler" ["id"]=> int(2) ["state"]=> string(7) "WAITING" } }

Kevinrasf · « **Reply #170 on:** June 03, 2015, 08:15:43 pm »

Quote from: lyaffe on June 01, 2015, 04:57:09 pm

You can open a command line window then invoke run.bat from there to get a better error message.
Most likely, you need to install Java 8.

Thanks alot it was the old Java version I got.

Updated to 8 and it runs now.

thx

Breasal · « **Reply #171 on:** June 06, 2015, 07:13:42 am »

Apologies if my question has been answered...did not read the full thread:

My assets are not display on the NRS, just getting the "thinking circle of never-ending spinning". Also, the dashboard does not display estimated value for my assets as before.

Is there a fix for this?

Thanks!

Switch to ArdorForum

News:

Author Topic: NRS v1.5.9 (Read 50364 times)

Riker

Re: NRS v1.5.9

th4o

Re: NRS v1.5.9

Tosch110

Re: NRS v1.5.9

davethetrousers

Re: NRS v1.5.9

ScripterRon

Re: NRS v1.5.9

ScripterRon

Re: NRS v1.5.9

Tosch110

Re: NRS v1.5.9

Riker

Re: NRS v1.5.9

coinomat

Re: NRS v1.5.9

Tosch110

Re: NRS v1.5.9

Kevinrasf

Re: NRS v1.5.9

Breasal

Re: NRS v1.5.9