Server is out of disk space error

Message boards : Number crunching : Server is out of disk space error
Message board moderation

To post messages, you must log in.

AuthorMessage
biodoc

Send message
Joined: 10 Apr 21
Posts: 5
Credit: 9,635,544
RAC: 204,378
Message 310 - Posted: 25 Apr 2022, 22:26:34 UTC

Mon 25 Apr 2022 06:24:16 PM EDT | Gaia@home | [error] Error reported by file upload server: Server is out of disk space
ID: 310 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
gemini8
Avatar

Send message
Joined: 8 Oct 20
Posts: 7
Credit: 1,109,540
RAC: 6,951
Message 329 - Posted: 20 Jul 2022, 12:20:37 UTC

And once again:
Mi 20 Jul 2022 14:16:51 CEST | Gaia@home | [error] Error reported by file upload server: Server is out of disk space

- - - - - - - - - -
Greetings, Jens
ID: 329 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Conan
Avatar

Send message
Joined: 27 Apr 20
Posts: 40
Credit: 2,829,759
RAC: 58,401
Message 330 - Posted: 20 Jul 2022, 21:49:31 UTC
Last modified: 20 Jul 2022, 21:53:35 UTC

Out of disk space again, can't upload results.

Conan

PS Just after I post it starts working again.
ID: 330 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Conan
Avatar

Send message
Joined: 27 Apr 20
Posts: 40
Credit: 2,829,759
RAC: 58,401
Message 331 - Posted: 21 Jul 2022, 0:20:00 UTC

Back again, "No Space Left On Server"

unable to upload as "Can't write to disk"

Conan
ID: 331 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Saenger

Send message
Joined: 13 Feb 20
Posts: 2
Credit: 1,239,269
RAC: 5,822
Message 332 - Posted: 21 Jul 2022, 14:10:07 UTC

And:
Do 21 Jul 2022 16:08:38 CEST | Gaia@home | Scheduler request completed: got 0 new tasks
Do 21 Jul 2022 16:08:38 CEST | Gaia@home | Project is temporarily shut down for maintenance
ID: 332 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Conan
Avatar

Send message
Joined: 27 Apr 20
Posts: 40
Credit: 2,829,759
RAC: 58,401
Message 333 - Posted: 26 Jul 2022, 2:36:20 UTC

Server out of Disk Space , can't upload work.

Conan
ID: 333 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Saenger

Send message
Joined: 13 Feb 20
Posts: 2
Credit: 1,239,269
RAC: 5,822
Message 334 - Posted: 26 Jul 2022, 8:18:05 UTC

Yep:
Di 26 Jul 2022 10:03:43 CEST | Gaia@home | [fxd] starting upload, upload_offset -1
Di 26 Jul 2022 10:03:43 CEST | Gaia@home | Started upload of 10_1043379_0_r1196165969_0
Di 26 Jul 2022 10:03:43 CEST | Gaia@home | [file_xfer] URL: http://150.254.66.104/gaiaathome_cgi/file_upload_handler
Di 26 Jul 2022 10:03:44 CEST | Gaia@home | [file_xfer] http op done; retval 0 (Success)
Di 26 Jul 2022 10:03:44 CEST | Gaia@home | [error] Error reported by file upload server: Server is out of disk space
Di 26 Jul 2022 10:03:44 CEST | Gaia@home | [file_xfer] parsing upload response: <data_server_reply>    <status>1</status>    <message>Server is out of disk space</message></data_server_reply>
Di 26 Jul 2022 10:03:44 CEST | Gaia@home | [file_xfer] parsing status: -127
Di 26 Jul 2022 10:03:44 CEST | Gaia@home | [file_xfer] file transfer status -127 (transient upload error)
Di 26 Jul 2022 10:03:44 CEST | Gaia@home | Temporarily failed upload of 10_1043379_0_r1196165969_0: transient upload error
Di 26 Jul 2022 10:03:44 CEST | Gaia@home | [file_xfer] project-wide upload delay for 1106.194003 sec
Di 26 Jul 2022 10:03:44 CEST | Gaia@home | Backing off 00:13:50 on upload of 10_1043379_0_r1196165969_0

ID: 334 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Coleslaw
Avatar

Send message
Joined: 6 Sep 19
Posts: 2
Credit: 1,138,489
RAC: 9,833
Message 337 - Posted: 4 Aug 2022, 23:26:48 UTC - in response to Message 334.  

Yeah..it is insanely annoying to have the client constantly backing off from uploading. The second it does upload something, I can pull new work. Yes I know that is because of the server side limits that are in place and yes I know how to side step it in a way. However, it would be nice to just have this issue rectified as it has gone on long enough. An admin could at least chime in to acknowledge the issue.
ID: 337 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
pututu
Avatar

Send message
Joined: 25 Sep 19
Posts: 6
Credit: 5,974,417
RAC: 31,306
Message 339 - Posted: 13 Aug 2022, 14:24:02 UTC

Sat 13 Aug 2022 07:22:01 AM PDT | Gaia@home | Started upload of 10_3572576_0_r1070299446_0
Sat 13 Aug 2022 07:22:02 AM PDT | Gaia@home | [error] Error reported by file upload server: Server is out of disk space
Sat 13 Aug 2022 07:22:02 AM PDT | Gaia@home | Temporarily failed upload of 10_3572576_0_r1070299446_0: transient upload error
Sat 13 Aug 2022 07:22:02 AM PDT | Gaia@home | Backing off 00:03:49 on upload of 10_3572576_0_r1070299446_0


Server is out of disk space again for the nth time.....
ID: 339 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
pututu
Avatar

Send message
Joined: 25 Sep 19
Posts: 6
Credit: 5,974,417
RAC: 31,306
Message 340 - Posted: 13 Aug 2022, 16:12:20 UTC
Last modified: 13 Aug 2022, 16:28:18 UTC

Each task is at least 6MB in file size during uploading. If I finish 1000 tasks a day, that's 6GB of data been uploaded. Since the run time for each task is relatively short, let's assume say 5 mins. On a 16C/32T system running 100%, that will mean in a day, about 9,216 tasks are completed. This is equivalent to about 55GB of data been transferred. Assuming that 55GB of data stays in the hard drive for sometime, (maybe less after db assimilation, don't know), you really need a lot of disk space.

Looking at the server status just before the disk space issue started, I'm seeing about 700 tasks in progress, meaning about 700 threads running at any one time. This is roughly equivalent to about 1.2 TB of data been uploaded per day.

I think this project needs a lot of more server disk space or more frequent server maintenance (aka downtime) to compress db, post-processing or what not. My 2 cents.
ID: 340 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Conan
Avatar

Send message
Joined: 27 Apr 20
Posts: 40
Credit: 2,829,759
RAC: 58,401
Message 341 - Posted: 14 Aug 2022, 7:09:15 UTC

Still can't upload due to server out of disk space.

Conan
ID: 341 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
zupa

Send message
Joined: 21 Aug 19
Posts: 110
Credit: 888,695
RAC: 8,066
Message 342 - Posted: 14 Aug 2022, 11:23:55 UTC - in response to Message 341.  

fixed again ....
ID: 342 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Conan
Avatar

Send message
Joined: 27 Apr 20
Posts: 40
Credit: 2,829,759
RAC: 58,401
Message 343 - Posted: 14 Aug 2022, 11:50:50 UTC

Thanks zupa.

Awaiting server to come back on line to upload, project requesting 3600 second delay, and server is off line.

Conan
ID: 343 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
kotenok2000

Send message
Joined: 12 Nov 20
Posts: 6
Credit: 16,804
RAC: 512
Message 344 - Posted: 16 Aug 2022, 0:55:59 UTC
Last modified: 16 Aug 2022, 1:08:42 UTC

Can they set boinc to gzip results before uploading?
ID: 344 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
kotenok2000

Send message
Joined: 12 Nov 20
Posts: 6
Credit: 16,804
RAC: 512
Message 345 - Posted: 16 Aug 2022, 17:52:30 UTC
Last modified: 16 Aug 2022, 18:44:51 UTC

Looks like validation code is very slow 2695 workunits in 2 hours.
https://grafana.kiska.pw/d/boinc/boinc?orgId=1&var-project=gaia@home&from=now-6h&to=now&refresh=1m
ID: 345 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
zupa

Send message
Joined: 21 Aug 19
Posts: 110
Credit: 888,695
RAC: 8,066
Message 346 - Posted: 16 Aug 2022, 20:01:59 UTC - in response to Message 345.  

Yes, I see this problem with validation process ....
ID: 346 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Coleslaw
Avatar

Send message
Joined: 6 Sep 19
Posts: 2
Credit: 1,138,489
RAC: 9,833
Message 347 - Posted: 16 Aug 2022, 23:07:57 UTC

My work units seem to be around 7 minutes as well. Is there any way to pack multiple work units together so that the run time is longer? Not sure if it would reduce any files in the process but the amount of data being transferred is a bit crazy for what little running time they have. My home internet has a data limit and this project would easily chew through it in no time.
ID: 347 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
kotenok2000

Send message
Joined: 12 Nov 20
Posts: 6
Credit: 16,804
RAC: 512
Message 348 - Posted: 17 Aug 2022, 1:48:04 UTC - in response to Message 347.  

Validation backlog is empty but no new tasks.
ID: 348 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote

Message boards : Number crunching : Server is out of disk space error

©2024 GAVIP-GC