Java protobuf creating too many objectsIs Java “pass-by-reference” or “pass-by-value”?How do I efficiently iterate over each entry in a Java Map?Does a finally block always get executed in Java?Create ArrayList from arrayWhat is the difference between public, protected, package-private and private in Java?How do I read / convert an InputStream into a String in Java?When to use LinkedList over ArrayList in Java?How do I generate random integers within a specific range in Java?How do I convert a String to an int in Java?Creating a memory leak with Java
Journal losing indexing services
We have a love-hate relationship
A Permanent Norse Presence in America
Global amount of publications over time
Proving a function is onto where f(x)=|x|.
How do you respond to a colleague from another team when they're wrongly expecting that you'll help them?
Some numbers are more equivalent than others
Open a doc from terminal, but not by its name
Is it possible to use .desktop files to open local pdf files on specific pages with a browser?
anything or something to eat
Fuse symbol on toroidal transformer
Divine apple island
Longest common substring in linear time
Do Legal Documents Require Signing In Standard Pen Colors?
How much character growth crosses the line into breaking the character
What's the difference between 違法 and 不法?
Is XSS in canonical link possible?
Drawing a topological "handle" with Tikz
Is a model fitted to data or is data fitted to a model?
What linear sensor for a keyboard?
Query about absorption line spectra
Greco-Roman egalitarianism
How must one send away the mother bird?
Can the Supreme Court overturn an impeachment?
Java protobuf creating too many objects
Is Java “pass-by-reference” or “pass-by-value”?How do I efficiently iterate over each entry in a Java Map?Does a finally block always get executed in Java?Create ArrayList from arrayWhat is the difference between public, protected, package-private and private in Java?How do I read / convert an InputStream into a String in Java?When to use LinkedList over ArrayList in Java?How do I generate random integers within a specific range in Java?How do I convert a String to an int in Java?Creating a memory leak with Java
I have a jetty server proxyfying betweena backend and converting protobufs frontend.proto->backend.proto and backend.proto->frontend.proto.
Requests take below 20ms(99th) and 40ms(99.9th) while load is not peaking.
However, when load peaks, the 99th increases +10 but the 99.9th increases by +60.
I have investigated it and the delayed requests are caused by GC Evacuation pauses, this I am sure of, this pauses take 50-70ms and run once every 15 seconds on valley load but jump up to once every 3-5 seconds on peak load, the duration is the same.
As soon as the GC frequency goes below 8-9 seconds the 99.9th percentile shoots up and I can see the debug logs of slow requests concurrently with the GC log.
I have profiled with JProfiler, Yourkit and VisualVM and saw that:
- Eden space fills up and a GC pause is triggered
- Very few of the objects are moved into Survivor(few MB out of 12G)
- So most of the objects are already expired in Eden
- This makes sense since requests take 30-40ms and most of the objects lifetime is tied to the request lifetime
- Most of the objects are created during protobuf deserialization time
I've tried playing with GCPauseMillis and Eden sizes but nothing seems to make a difference, it never takes less than 50ms and bigger Eden means less frequency but much longer pauses
I see 2 options here:
- Somehow reuse object creation in java-protobuf: Seems impossible, read through a lot of posts and mails and its not setup that way, they just say that "Java object allocation is very efficient and it should be able to handle many objects being created", and while that is true, the associated GC cost is killing my 99.9th
- Make the GC run more often, say once a second, to bring collection time down so that it will stop more requests but for shorter times: I have been playing with GCMaxMillis and Eden sizes but I can't seem to get it lower than that
I uploaded the gc log to gc_log
Java version:
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)
GC details:
-XX:CICompilerCount=12 -XX:ConcGCThreads=5 -XX:ErrorFile=/home/y/var/crash/hs_err_pid%p.log -XX:+FlightRecorder -XX:G1HeapRegionSize=4194304 -XX:GCLogFileSize=4194304 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/y/logs/yjava_jetty -XX:InitialHeapSize=12884901888 -XX:MarkStackSize=4194304 -XX:MaxHeapSize=12884901888 -XX:MaxNewSize=7730102272 -XX:MinHeapDeltaBytes=4194304 -XX:NumberOfGCLogFiles=10 -XX:+ParallelRefProcEnabled -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UnlockCommercialFeatures -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseFastUnorderedTimeStamps -XX:+UseG1GC -XX:+UseGCLogFileRotation
java garbage-collection protocol-buffers
asked Mar 7 at 8:58
Arkaitz JimenezArkaitz Jimenez
16.1k86395
|
show 2 more comments
I have a jetty server proxyfying betweena backend and converting protobufs frontend.proto->backend.proto and backend.proto->frontend.proto.
Requests take below 20ms(99th) and 40ms(99.9th) while load is not peaking.
However, when load peaks, the 99th increases +10 but the 99.9th increases by +60.
I have investigated it and the delayed requests are caused by GC Evacuation pauses, this I am sure of, this pauses take 50-70ms and run once every 15 seconds on valley load but jump up to once every 3-5 seconds on peak load, the duration is the same.
As soon as the GC frequency goes below 8-9 seconds the 99.9th percentile shoots up and I can see the debug logs of slow requests concurrently with the GC log.
I have profiled with JProfiler, Yourkit and VisualVM and saw that:
- Eden space fills up and a GC pause is triggered
- Very few of the objects are moved into Survivor(few MB out of 12G)
- So most of the objects are already expired in Eden
- This makes sense since requests take 30-40ms and most of the objects lifetime is tied to the request lifetime
- Most of the objects are created during protobuf deserialization time
I've tried playing with GCPauseMillis and Eden sizes but nothing seems to make a difference, it never takes less than 50ms and bigger Eden means less frequency but much longer pauses
I see 2 options here:
- Somehow reuse object creation in java-protobuf: Seems impossible, read through a lot of posts and mails and its not setup that way, they just say that "Java object allocation is very efficient and it should be able to handle many objects being created", and while that is true, the associated GC cost is killing my 99.9th
- Make the GC run more often, say once a second, to bring collection time down so that it will stop more requests but for shorter times: I have been playing with GCMaxMillis and Eden sizes but I can't seem to get it lower than that
I uploaded the gc log to gc_log
Java version:
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)
GC details:
-XX:CICompilerCount=12 -XX:ConcGCThreads=5 -XX:ErrorFile=/home/y/var/crash/hs_err_pid%p.log -XX:+FlightRecorder -XX:G1HeapRegionSize=4194304 -XX:GCLogFileSize=4194304 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/y/logs/yjava_jetty -XX:InitialHeapSize=12884901888 -XX:MarkStackSize=4194304 -XX:MaxHeapSize=12884901888 -XX:MaxNewSize=7730102272 -XX:MinHeapDeltaBytes=4194304 -XX:NumberOfGCLogFiles=10 -XX:+ParallelRefProcEnabled -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UnlockCommercialFeatures -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseFastUnorderedTimeStamps -XX:+UseG1GC -XX:+UseGCLogFileRotation
java garbage-collection protocol-buffers
asked Mar 7 at 8:58
Arkaitz JimenezArkaitz Jimenez
16.1k86395
Can You add the details of which Jvm and garbage collector You are using to the question.
– Silver Shroud
Mar 7 at 16:11
@SilverShroud done!
– Arkaitz Jimenez
Mar 8 at 9:32
Checking the log output, object copy seems to cause the spike. But survivor or old generation size does not seem to increase in these cases(maybe a bug). I would try updated versions of java 8 runtime to see if it behaves the same. You can also try to play with following values to limit the size of eden space -XX:G1NewSizePercent -XX:G1MaxNewSizePercent
– Silver Shroud
Mar 8 at 16:17
@SilverShroud I've done some more testing and touching the G1 knobs does not seem to work, switching to CMS did though, latency dropped a lot because Eden Evacuation does not seem to pause with CMS. It does feel a bit backwards to use CMS while G1 is much more modern.
– Arkaitz Jimenez
Mar 8 at 16:20
I think these flags are experimental flags. To be able to use them some other flag has to be set. (look into this If You have missed it)
– Silver Shroud
Mar 8 at 16:22
|
show 2 more comments
I have a jetty server proxyfying betweena backend and converting protobufs frontend.proto->backend.proto and backend.proto->frontend.proto.
Requests take below 20ms(99th) and 40ms(99.9th) while load is not peaking.
However, when load peaks, the 99th increases +10 but the 99.9th increases by +60.
I have investigated it and the delayed requests are caused by GC Evacuation pauses, this I am sure of, this pauses take 50-70ms and run once every 15 seconds on valley load but jump up to once every 3-5 seconds on peak load, the duration is the same.
As soon as the GC frequency goes below 8-9 seconds the 99.9th percentile shoots up and I can see the debug logs of slow requests concurrently with the GC log.
I have profiled with JProfiler, Yourkit and VisualVM and saw that:
- Eden space fills up and a GC pause is triggered
- Very few of the objects are moved into Survivor(few MB out of 12G)
- So most of the objects are already expired in Eden
- This makes sense since requests take 30-40ms and most of the objects lifetime is tied to the request lifetime
- Most of the objects are created during protobuf deserialization time
I've tried playing with GCPauseMillis and Eden sizes but nothing seems to make a difference, it never takes less than 50ms and bigger Eden means less frequency but much longer pauses
I see 2 options here:
- Somehow reuse object creation in java-protobuf: Seems impossible, read through a lot of posts and mails and its not setup that way, they just say that "Java object allocation is very efficient and it should be able to handle many objects being created", and while that is true, the associated GC cost is killing my 99.9th
- Make the GC run more often, say once a second, to bring collection time down so that it will stop more requests but for shorter times: I have been playing with GCMaxMillis and Eden sizes but I can't seem to get it lower than that
I uploaded the gc log to gc_log
Java version:
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)
GC details:
-XX:CICompilerCount=12 -XX:ConcGCThreads=5 -XX:ErrorFile=/home/y/var/crash/hs_err_pid%p.log -XX:+FlightRecorder -XX:G1HeapRegionSize=4194304 -XX:GCLogFileSize=4194304 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/y/logs/yjava_jetty -XX:InitialHeapSize=12884901888 -XX:MarkStackSize=4194304 -XX:MaxHeapSize=12884901888 -XX:MaxNewSize=7730102272 -XX:MinHeapDeltaBytes=4194304 -XX:NumberOfGCLogFiles=10 -XX:+ParallelRefProcEnabled -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UnlockCommercialFeatures -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseFastUnorderedTimeStamps -XX:+UseG1GC -XX:+UseGCLogFileRotation
java garbage-collection protocol-buffers
asked Mar 7 at 8:58
Arkaitz JimenezArkaitz Jimenez
16.1k86395
I have a jetty server proxyfying betweena backend and converting protobufs frontend.proto->backend.proto and backend.proto->frontend.proto.
Requests take below 20ms(99th) and 40ms(99.9th) while load is not peaking.
However, when load peaks, the 99th increases +10 but the 99.9th increases by +60.
I have investigated it and the delayed requests are caused by GC Evacuation pauses, this I am sure of, this pauses take 50-70ms and run once every 15 seconds on valley load but jump up to once every 3-5 seconds on peak load, the duration is the same.
As soon as the GC frequency goes below 8-9 seconds the 99.9th percentile shoots up and I can see the debug logs of slow requests concurrently with the GC log.
I have profiled with JProfiler, Yourkit and VisualVM and saw that:
- Eden space fills up and a GC pause is triggered
- Very few of the objects are moved into Survivor(few MB out of 12G)
- So most of the objects are already expired in Eden
- This makes sense since requests take 30-40ms and most of the objects lifetime is tied to the request lifetime
- Most of the objects are created during protobuf deserialization time
I've tried playing with GCPauseMillis and Eden sizes but nothing seems to make a difference, it never takes less than 50ms and bigger Eden means less frequency but much longer pauses
I see 2 options here:
- Somehow reuse object creation in java-protobuf: Seems impossible, read through a lot of posts and mails and its not setup that way, they just say that "Java object allocation is very efficient and it should be able to handle many objects being created", and while that is true, the associated GC cost is killing my 99.9th
- Make the GC run more often, say once a second, to bring collection time down so that it will stop more requests but for shorter times: I have been playing with GCMaxMillis and Eden sizes but I can't seem to get it lower than that
I uploaded the gc log to gc_log
Java version:
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)
GC details:
-XX:CICompilerCount=12 -XX:ConcGCThreads=5 -XX:ErrorFile=/home/y/var/crash/hs_err_pid%p.log -XX:+FlightRecorder -XX:G1HeapRegionSize=4194304 -XX:GCLogFileSize=4194304 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/y/logs/yjava_jetty -XX:InitialHeapSize=12884901888 -XX:MarkStackSize=4194304 -XX:MaxHeapSize=12884901888 -XX:MaxNewSize=7730102272 -XX:MinHeapDeltaBytes=4194304 -XX:NumberOfGCLogFiles=10 -XX:+ParallelRefProcEnabled -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UnlockCommercialFeatures -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseFastUnorderedTimeStamps -XX:+UseG1GC -XX:+UseGCLogFileRotation
java garbage-collection protocol-buffers
java garbage-collection protocol-buffers
asked Mar 7 at 8:58
Arkaitz JimenezArkaitz Jimenez
16.1k86395
asked Mar 7 at 8:58
Arkaitz JimenezArkaitz Jimenez
16.1k86395
edited Mar 8 at 9:32
Arkaitz Jimenez
asked Mar 7 at 8:58
Arkaitz JimenezArkaitz Jimenez
16.1k86395
asked Mar 7 at 8:58
Arkaitz JimenezArkaitz Jimenez
16.1k86395
asked Mar 7 at 8:58
Arkaitz JimenezArkaitz Jimenez
16.1k86395
16.1k86395
Can You add the details of which Jvm and garbage collector You are using to the question.
– Silver Shroud
Mar 7 at 16:11
@SilverShroud done!
– Arkaitz Jimenez
Mar 8 at 9:32
Checking the log output, object copy seems to cause the spike. But survivor or old generation size does not seem to increase in these cases(maybe a bug). I would try updated versions of java 8 runtime to see if it behaves the same. You can also try to play with following values to limit the size of eden space -XX:G1NewSizePercent -XX:G1MaxNewSizePercent
– Silver Shroud
Mar 8 at 16:17
@SilverShroud I've done some more testing and touching the G1 knobs does not seem to work, switching to CMS did though, latency dropped a lot because Eden Evacuation does not seem to pause with CMS. It does feel a bit backwards to use CMS while G1 is much more modern.
– Arkaitz Jimenez
Mar 8 at 16:20
I think these flags are experimental flags. To be able to use them some other flag has to be set. (look into this If You have missed it)
– Silver Shroud
Mar 8 at 16:22
|
show 2 more comments
Can You add the details of which Jvm and garbage collector You are using to the question.
– Silver Shroud
Mar 7 at 16:11
@SilverShroud done!
– Arkaitz Jimenez
Mar 8 at 9:32
Checking the log output, object copy seems to cause the spike. But survivor or old generation size does not seem to increase in these cases(maybe a bug). I would try updated versions of java 8 runtime to see if it behaves the same. You can also try to play with following values to limit the size of eden space -XX:G1NewSizePercent -XX:G1MaxNewSizePercent
– Silver Shroud
Mar 8 at 16:17
@SilverShroud I've done some more testing and touching the G1 knobs does not seem to work, switching to CMS did though, latency dropped a lot because Eden Evacuation does not seem to pause with CMS. It does feel a bit backwards to use CMS while G1 is much more modern.
– Arkaitz Jimenez
Mar 8 at 16:20
I think these flags are experimental flags. To be able to use them some other flag has to be set. (look into this If You have missed it)
– Silver Shroud
Mar 8 at 16:22
Can You add the details of which Jvm and garbage collector You are using to the question.
– Silver Shroud
Mar 7 at 16:11
Can You add the details of which Jvm and garbage collector You are using to the question.
– Silver Shroud
Mar 7 at 16:11
@SilverShroud done!
– Arkaitz Jimenez
Mar 8 at 9:32
@SilverShroud done!
– Arkaitz Jimenez
Mar 8 at 9:32
Checking the log output, object copy seems to cause the spike. But survivor or old generation size does not seem to increase in these cases(maybe a bug). I would try updated versions of java 8 runtime to see if it behaves the same. You can also try to play with following values to limit the size of eden space -XX:G1NewSizePercent -XX:G1MaxNewSizePercent
– Silver Shroud
Mar 8 at 16:17
Checking the log output, object copy seems to cause the spike. But survivor or old generation size does not seem to increase in these cases(maybe a bug). I would try updated versions of java 8 runtime to see if it behaves the same. You can also try to play with following values to limit the size of eden space -XX:G1NewSizePercent -XX:G1MaxNewSizePercent
– Silver Shroud
Mar 8 at 16:17
@SilverShroud I've done some more testing and touching the G1 knobs does not seem to work, switching to CMS did though, latency dropped a lot because Eden Evacuation does not seem to pause with CMS. It does feel a bit backwards to use CMS while G1 is much more modern.
– Arkaitz Jimenez
Mar 8 at 16:20
@SilverShroud I've done some more testing and touching the G1 knobs does not seem to work, switching to CMS did though, latency dropped a lot because Eden Evacuation does not seem to pause with CMS. It does feel a bit backwards to use CMS while G1 is much more modern.
– Arkaitz Jimenez
Mar 8 at 16:20
I think these flags are experimental flags. To be able to use them some other flag has to be set. (look into this If You have missed it)
– Silver Shroud
Mar 8 at 16:22
I think these flags are experimental flags. To be able to use them some other flag has to be set. (look into this If You have missed it)
– Silver Shroud
Mar 8 at 16:22
|
show 2 more comments
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55039713%2fjava-protobuf-creating-too-many-objects%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55039713%2fjava-protobuf-creating-too-many-objects%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Can You add the details of which Jvm and garbage collector You are using to the question.
– Silver Shroud
Mar 7 at 16:11
@SilverShroud done!
– Arkaitz Jimenez
Mar 8 at 9:32
Checking the log output, object copy seems to cause the spike. But survivor or old generation size does not seem to increase in these cases(maybe a bug). I would try updated versions of java 8 runtime to see if it behaves the same. You can also try to play with following values to limit the size of eden space -XX:G1NewSizePercent -XX:G1MaxNewSizePercent
– Silver Shroud
Mar 8 at 16:17
@SilverShroud I've done some more testing and touching the G1 knobs does not seem to work, switching to CMS did though, latency dropped a lot because Eden Evacuation does not seem to pause with CMS. It does feel a bit backwards to use CMS while G1 is much more modern.
– Arkaitz Jimenez
Mar 8 at 16:20
I think these flags are experimental flags. To be able to use them some other flag has to be set. (look into this If You have missed it)
– Silver Shroud
Mar 8 at 16:22