From d9604d9258ce29d955d89ce3f6d72dfd5119a42a Mon Sep 17 00:00:00 2001 From: tivrfoa Date: Sat, 27 Jan 2024 15:41:00 -0300 Subject: Use LinkedBlockingQueue to process results - based on thomaswue (#603) /** * Solution based on thomaswue solution, commit: * commit d0a28599c293d3afe3291fc3cf169a7b25ae9ae6 * Author: Thomas Wuerthinger * Date: Sun Jan 21 20:13:48 2024 +0100 * * Changes: * 1) Use LinkedBlockingQueue to store partial results, that * will then be merged into the final map later. * As different chunks finish at different times, this allows * to process them as they finish, instead of joining the * threads sequentially. * This change seems more useful for the 10k dataset, as the * runtime difference of each chunk is greater. * 2) Use only 4 threads if the file is >= 14GB. * This showed much better results on my local test, but I only * run with 200 million rows (because of limited RAM), and I have * no idea how it will perform on the 1brc HW. */ --- calculate_average_tivrfoa.sh | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) create mode 100755 calculate_average_tivrfoa.sh (limited to 'calculate_average_tivrfoa.sh') diff --git a/calculate_average_tivrfoa.sh b/calculate_average_tivrfoa.sh new file mode 100755 index 0000000..cec66fd --- /dev/null +++ b/calculate_average_tivrfoa.sh @@ -0,0 +1,24 @@ +#!/bin/sh +# +# Copyright 2023 The original authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +if [ -f target/CalculateAverage_tivrfoa_image ]; then + target/CalculateAverage_tivrfoa_image +else + JAVA_OPTS="--enable-preview" + java $JAVA_OPTS --class-path target/average-1.0.0-SNAPSHOT.jar dev.morling.onebrc.CalculateAverage_tivrfoa +fi + -- cgit v1.2.3