persistent files

Message boards : Number crunching : persistent files
Message board moderation

To post messages, you must log in.

AuthorMessage
Sergey Kovalchuk

Send message
Joined: 21 Aug 19
Posts: 23
Credit: 91,637
RAC: 0
Message 3 - Posted: 6 Sep 2020, 11:43:48 UTC

a huge number of small tasks - a lot of time is spent downloading input files

while both input files are the same for this version of the application or at least the current series of tasks (batch)
you can significantly reduce the load on the server if you mark these files with the "sticky" tag in the WU template
<input_template>
    <file_info>
    . . . . . .
         <sticky/> 
         <no_delete/> 
    </file_info>

    <workunit>
    . . . . . .
    </workunit>
</input_template>


<sticky/> if present, the file remains on the client after job is finished.
<no_delete/> if present, the file is not deleted from the server after job is completed. Use this if the file is used as input to more than one job.


the option is possible -
constant config for the application version,
input data are constant for a series of tasks, a specific batch, etc.
ID: 3 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ChelseaOilman

Send message
Joined: 9 May 20
Posts: 2
Credit: 620,446
RAC: 31,475
Message 4 - Posted: 6 Sep 2020, 17:51:20 UTC

Yes please!

And unless these are just test tasks to see if the server will hold up under load, please make longer tasks.
ID: 4 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
zupa

Send message
Joined: 21 Aug 19
Posts: 115
Credit: 888,695
RAC: 1,654
Message 13 - Posted: 7 Sep 2020, 21:58:14 UTC - in response to Message 4.  

Thank you for the information. Please be patient . These are test tasks for server ...
ID: 13 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Crystal Pellet

Send message
Joined: 29 Sep 20
Posts: 14
Credit: 64,341
RAC: 0
Message 77 - Posted: 9 Oct 2020, 16:32:28 UTC

I did 'only' 114 tasks on 1 PC so far and have already loaded 13GB Gaia-project data.
Only 3.5GB left for BOINC and I can't extend the space or move the data.
Since bin-files are not purged after the job, I've to reset the project regulary.
ID: 77 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
zupa

Send message
Joined: 21 Aug 19
Posts: 115
Credit: 888,695
RAC: 1,654
Message 78 - Posted: 10 Oct 2020, 7:01:35 UTC - in response to Message 77.  

I used option </nodelete> for limiting transfer, but problem with space.
So, I am thinking of a compromise ....
ID: 78 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote

Message boards : Number crunching : persistent files

©2024 GAVIP-GC