For large file download, aria2 supports -x 16 -s 32, options that are for multithreaded multichunk download.
Using python asyncio we can do this, by getting file size, and creating chunk parts from file size and parallel request each chunk to temp file and join them
Can same be done using curl/wget + bash +xargs/parallel utilities
Using
- curl head request and get total size
- create chunk ranges
- curl --range option to download each chunk in parallel (In latest curl -Z support parallel, or in old curl, use xargs -P or gnu parallel -J options)
- combine all parts
Can anyone help with creating bash function with failsafe - error handling and retry in case of sever load and empty chunk or temporary ip ban to ise while True loop until file has been succesfully downloaded
Can someone explain: in case of idm and aria2 like tools i have read that they downlaod by creating multiple connections to server how -x 16 and -s 32 options work. how creating multiple connection works for multithreaded parallel downloafing and how can we implememy it with curl + bash
