Imagine that you would like to burn your website to a set of DVDs (no controllers, no slowly-discharging flash chips, cheap price-per-pcs if mass distribution is intended, other benefits like this that are still relevant). You specify --warc-max-size=LIMIT, where LIMIT is a size of DVD minus some reasonable overhead for a filesystem etc. - i.e. 4278190080 bytes
Expected behaviour: all the WARC parts are either LIMIT bytes or somewhat smaller than this
Observed behaviour: all the WARC parts that have successfully reached the LIMIT bytes (no connection errors causing my grabber.sh script from #251 to terminate the grab-site processes and them restart them with --resume causing a premature creation of a new WARC part and continuing there) - end up LARGER than LIMIT bytes: 4279381231 , 4278853242 , etc.
What I suspect a logic here is: it keeps filling a WARC part until its size exceeds the LIMIT bytes, then it moves to a new part. But this logic is erroneous, a correct way of doing this is always checking if a " current WARC part size + downloaded file size + internal WARC header size " exceeds the LIMIT bytes - and, if yes, start a new WARC part and save that file there
Possible workaround: manually extract all the WARC parts with warcat, merge their contents, then repack manually with something like warcit --name website.warc.gz https://www.website.com/ ./www.website.com/ then somehow split this WARC into parts
Imagine that you would like to burn your website to a set of DVDs (no controllers, no slowly-discharging flash chips, cheap price-per-pcs if mass distribution is intended, other benefits like this that are still relevant). You specify --warc-max-size=LIMIT, where LIMIT is a size of DVD minus some reasonable overhead for a filesystem etc. - i.e. 4278190080 bytes
Expected behaviour: all the WARC parts are either LIMIT bytes or somewhat smaller than this
Observed behaviour: all the WARC parts that have successfully reached the LIMIT bytes (no connection errors causing my grabber.sh script from #251 to terminate the grab-site processes and them restart them with --resume causing a premature creation of a new WARC part and continuing there) - end up LARGER than LIMIT bytes: 4279381231 , 4278853242 , etc.
What I suspect a logic here is: it keeps filling a WARC part until its size exceeds the LIMIT bytes, then it moves to a new part. But this logic is erroneous, a correct way of doing this is always checking if a " current WARC part size + downloaded file size + internal WARC header size " exceeds the LIMIT bytes - and, if yes, start a new WARC part and save that file there
Possible workaround: manually extract all the WARC parts with warcat, merge their contents, then repack manually with something like
warcit --name website.warc.gz https://www.website.com/ ./www.website.com/then somehow split this WARC into parts