Finding download information

Page 1 of 2 1, 2  Next

View previous topic View next topic Go down

Finding download information

Post  PGTips91 on Fri Apr 02, 2010 9:23 pm

Following on from my previous post regarding finding the information for the latest version to download, I wish to bring up a couple of ideas that I have been investigating since I downloaded a copy of the latest Sabayon and saw what they are doing there with how they are making their downloads available.

Here are two ideas worth incorporating with each new release : --

MetaLinks

Aria2

I have started a Wiki page to assemble information on this field of investigation, and although I'm finding it a bit heavy going, the prospects are real that this could considerably improve the downloading experience with large files like Linux ISOs.

http://pgtips91.pbworks.com/Metalinks-and-aria2

Although aria2 does not re-use locally held data in the way that zsync does, it does combine downloading from a single torrent with downloads from any number of http or ftp mirrors. This has the best-of-all-worlds effect of allowing the download to be checked and any chunk that is in error being downloaded, guaranteeing a good download every time - something that is of value to most people - and also, since it will upload to the torrent, this would allow a torrent to get started quickly once the links have been posted. Also anyone could start seeding from their completed download at any time needed later on.

Of course, if aria2 could be combined with zsync to reuse local data that would be an added benefit, but so far my attempts to do this have failed. I may pursue this by contacting the developers and putting this forward as a suggestion. The algorithm used by zsync is the same as that used by rsync and should be reasonably well known by those who are able to write these kinds of applications, so here's hoping.

Another positive with aria2 is that there are several front-ends in development as well as ongoing development of the CLI application itself - something that is not the case with zsync, unfortunately.

Meanwhile I will continue to experiment with and learn the use of aria2 and Metalinks.

There is a thread I stared on the Linux Internationals forum here
aria2 and Metalinks

Paul
avatar
PGTips91

Posts : 71
Points : 75
Reputation : 0
Join date : 2010-01-28
Age : 76
Location : Auckland, New Zealand

View user profile http://picasaweb.google.com/paulgtaylor91

Back to top Go down

Re: Finding download information

Post  PGTips91 on Fri Apr 02, 2010 10:17 pm

As a way to avoid having to upload the entire ISO to a web server before distributing it, which causes delays and the possibility of a corrupted file being transferred, it might be a good strategy just to start with a torrent.

I am reading up on how to make and host a .torrent file here : --

Brian's BitTorrent FAQ and Guide - creating a new torrent

Paul
avatar
PGTips91

Posts : 71
Points : 75
Reputation : 0
Join date : 2010-01-28
Age : 76
Location : Auckland, New Zealand

View user profile http://picasaweb.google.com/paulgtaylor91

Back to top Go down

Re: Finding download information

Post  PGTips91 on Fri Apr 09, 2010 1:15 am

As an addendum to what I wrote above, I have been experimenting with aria2, a CLI download tool, which will download from both a single torrent and any number of HTML or FTP servers at the same time. It is actually quite easy to use it, once you've done it a time or two, and it has a number of advantages.

1) It uses multiple threads to download, so it will maximise your download speed.
2) If it has a .torrent to download from, as well as an HTML or FTP URL, it will check the completed file for errors, chunk by chunk, eliminating the risk of a bad download and having to download a second time.
3) As soon as the download is completed it starts seeding the torrent, which could considerably speed the process up at the start of a torrent.
4) There are no limits on the number of mirrors that can be included.
5) Aria2 can handle metalink files, so that it can be fed all the torrents and URLs in one small download file.
6) You can create your own input file, with a list of URLs and feed that to Aria2

Unfortunately at present Aria2 cannot utilise an existing ISO file, the way zsync does, and I have not been able to get Aria2 to utilise the partial file that zsync outputs. This would be a useful feature to add to Aria2 but I haven't yet tried suggesting this to the project.

Two points :

1) Next release should have a metalink available.
2) Next release should also ideally have a torrent.

Finally, it would be great if a zsync file was also made available. Then everybody would have what they needed in order to download the new ISO file reliably and efficiently.

Paul
avatar
PGTips91

Posts : 71
Points : 75
Reputation : 0
Join date : 2010-01-28
Age : 76
Location : Auckland, New Zealand

View user profile http://picasaweb.google.com/paulgtaylor91

Back to top Go down

Re: Finding download information

Post  StanTheMan on Fri Apr 09, 2010 2:54 pm

I am on a high speed North American system , largely fibre-optics. But I have been unable to download the Flighthawk ISO from the RAC site. Tried several times and it just starts then stalls.
I have no problem downloading from other Debian or Ubuntu sources.

StanTheMan

Posts : 79
Points : 81
Reputation : 0
Join date : 2010-01-29
Age : 97
Location : Kamloops BC Canada

View user profile

Back to top Go down

Re: Finding download information

Post  Admin on Fri Apr 09, 2010 5:55 pm

StanTheMan wrote:I am on a high speed North American system , largely fibre-optics. But I have been unable to download the Flighthawk ISO from the RAC site. Tried several times and it just starts then stalls.
I have no problem downloading from other Debian or Ubuntu sources.

Odd if you send a prepaid cd mailer with a blank +R DVD I will be happy to mail it to you.

R.A.C. Computers
Attn: PhoenixOS
246 9th Street North
St. Petersburg, FL 33705

Ray
avatar
Admin
Admin

Posts : 720
Points : 2642
Reputation : 1
Join date : 2010-01-28

View user profile http://phoenixos.forumotion.com

Back to top Go down

Re: Finding download information

Post  PGTips91 on Fri Apr 09, 2010 7:30 pm

Just adding a bit more information : --


Click to go there
26 Mar : DownThemAll! 2.0 beta 1 (Firefox extension) is out!
26 Mar : DownThemAll! 1.1.9 (Firefox extension) is out! This is the easiest way to try out Metalink!
23 Mar : Announcing: FTP Extensions for Cryptographic Hashes IETF Internet Draft. This is a work in progress and not meant to be implemented, only reviewed.
12 Mar : aria2 1.9.0! (Unix/Win), the high speed command line download utility with Metalink support, is out! Congratulations and thanks to the project, which is now 4 years old!

There is an on-line metalink generator at : --

metalink generator

Just fill in the data and generate a metalink file to download and post for users with your announcement.



Paul
avatar
PGTips91

Posts : 71
Points : 75
Reputation : 0
Join date : 2010-01-28
Age : 76
Location : Auckland, New Zealand

View user profile http://picasaweb.google.com/paulgtaylor91

Back to top Go down

Re: Finding download information

Post  PGTips91 on Fri Apr 09, 2010 10:26 pm

I have just put together a small blog on metalinks and aria2, here : --


Distributing and Downloading ISO and other large files.


Paul
avatar
PGTips91

Posts : 71
Points : 75
Reputation : 0
Join date : 2010-01-28
Age : 76
Location : Auckland, New Zealand

View user profile http://picasaweb.google.com/paulgtaylor91

Back to top Go down

Re: Finding download information

Post  PGTips91 on Sat Apr 10, 2010 8:02 am

I have created a torrent for the beta release of PhoenixOS and it has been posted on my Wiki at : --

http://pgtips91.pbworks.com/f/phoenixos030410.iso.torrent

I have written a short how-to on using aria2 to download from both this torrent and the main server at

http://www.raccomputers.com/phoenixos030410.iso

The tutorial is at How to get PhoenixOS aria2 metalinks

Basically all it takes is to get into the right directory in Konsole and type these commands : --

Code:
wget -c  http://pgtips91.pbworks.com/f/phoenixos030410.iso.torrent
and then,

Code:
aria2c -T phoenixos030410.iso.torrent  http://www.raccomputers.com/phoenixos030410.iso

and this will combine both the torrent and the server, checking that the download is valid and giving you the best download speed available.

Aria2 should be available in the repositories with Synaptic.

Anyone game to try it out?

I'm off to bed but the torrent is being seeded in KTorrent. I will check up on this in the morning.

Paul
avatar
PGTips91

Posts : 71
Points : 75
Reputation : 0
Join date : 2010-01-28
Age : 76
Location : Auckland, New Zealand

View user profile http://picasaweb.google.com/paulgtaylor91

Back to top Go down

Re: Finding download information

Post  Deffi on Sat Apr 10, 2010 12:59 pm

Have tried the above method and have a successful download. It was not quick but at present the internet is busy. So thinking about that, it is quicker than a straight download and is seeding a download somewhere.

David.
avatar
Deffi

Posts : 106
Points : 126
Reputation : 0
Join date : 2010-01-30
Age : 76
Location : UK

View user profile

Back to top Go down

Re: Finding download information

Post  PGTips91 on Sat Apr 10, 2010 6:32 pm

Deffi wrote:Have tried the above method and have a successful download. It was not quick but at present the internet is busy. So thinking about that, it is quicker than a straight download and is seeding a download somewhere.

David.

Yeah, that's about my experience.

Overnight I downloaded, as an experiment, a version of Calculate Linux just to see how my broadband connection would perform as well as learning some more about aria2.

When I started the download it was so slow that the ETA was days away, but when I got up this morning and checked on progress it had finished downloading the file and was seeding it.

Code:
Final results : --

-----------------------------------------------------------------------------------------------------------------------------------------

[#1 SEEDING(ratio:0.1) CN:0 UP:0.00KiB/s(247.4MiB)]
2010-04-11 10:02:59.279802 NOTICE - Shutdown sequence commencing... Press Ctrl-C again for emergency shutdown.
[#1 SEEDING(ratio:0.1) CN:0 UP:0.00KiB/s(247.4MiB)]
Download Results:
gid|stat|avg speed  |path/URI
===+====+===========+==========================================================
  1|  OK|  92.9KiB/s|./cld-10.4-i686.iso

Status Legend:
 (OK):download completed.

real    590m26.342s
user    0m29.098s
sys    0m11.529s
paul@paul-ArtistX07:/media/Elements_/LinuxDownloads/DownloadedISOfiles/CalculateLinux$

Since my broadband is choked for torrents during normal hours, this is quite satisfactory for me.

Using aria2 and a combination of the torrent and server(s) is a pretty good solution as it allows for automatic error-checking during the download process and guaranteeing a good result each time as well as optimising the download speed. It is just a pity that it is only a CLI tool and that will put some people off using it. It is pretty simple to use, though, once you get over the fear of the CLI.

Notice especially the very low use of resources - in nearly ten hours of operation aria2 consumed just over one minute of CPU cycles! Use of RAM memory is not large, either, so you can just start a download and continue to use your computer for other things, leaving this to run in the background until completion whenever that will be.

Paul
avatar
PGTips91

Posts : 71
Points : 75
Reputation : 0
Join date : 2010-01-28
Age : 76
Location : Auckland, New Zealand

View user profile http://picasaweb.google.com/paulgtaylor91

Back to top Go down

Re: Finding download information

Post  PGTips91 on Mon Apr 12, 2010 11:16 pm

I would be obliged if someone could check to see if my torrents are available for downloading.

After I started seeding the field for "Available" seems to be empty even thought the "Downloaded" field is full. I tried 'Importing' but it still shows the same.

At least, if you download the .torrent file and use aria2 you are more or less guaranteed a verified download even if it all comes from the server. But I would like to know that the file is available as a torrent as well.

Paul
avatar
PGTips91

Posts : 71
Points : 75
Reputation : 0
Join date : 2010-01-28
Age : 76
Location : Auckland, New Zealand

View user profile http://picasaweb.google.com/paulgtaylor91

Back to top Go down

Re: Finding download information

Post  StanTheMan on Mon Apr 12, 2010 11:56 pm

Paul.

I hamy torrent client running. I get your ISO listing at 1.4 GB
None dowwnloaded.
Downloading from 0 of 0 peers - idle .

My locastion is Canada.

StanTheMan

Posts : 79
Points : 81
Reputation : 0
Join date : 2010-01-29
Age : 97
Location : Kamloops BC Canada

View user profile

Back to top Go down

Re: Finding download information

Post  StanTheMan on Tue Apr 13, 2010 12:17 am

Paul.

My torrent has started downloading.
Now downloaded 316 KB.
Downloading from 1 ot 1 connected peers.

I will leave it running all night ( my night)

.

StanTheMan

Posts : 79
Points : 81
Reputation : 0
Join date : 2010-01-29
Age : 97
Location : Kamloops BC Canada

View user profile

Back to top Go down

Re: Finding download information

Post  StanTheMan on Tue Apr 13, 2010 5:58 pm

Paul.

I let my torrent run for 11 hours. Came bvack, and it shows downloaded 99 % , and now idle.
But I checked my file manager and I have an ISO of 1418 MB appears to be complete.
I burned this. The disk gives a boot prompt , and error ' cannot find kernel image ' .

StanTheMan

Posts : 79
Points : 81
Reputation : 0
Join date : 2010-01-29
Age : 97
Location : Kamloops BC Canada

View user profile

Back to top Go down

Re: Finding download information

Post  PGTips91 on Tue Apr 13, 2010 7:19 pm

StanTheMan wrote:Paul.

I let my torrent run for 11 hours. Came back, and it shows downloaded 99 % , and now idle.
But I checked my file manager and I have an ISO of 1418 MB appears to be complete.
I burned this. The disk gives a boot prompt , and error ' cannot find kernel image ' .

Ok, thanks for the feedback. I don't know what is going on here. I didn't see any activity at my end and after a long time I stopped the process by rebooting in order to reclaim memory so I could run VirtualBox which was refusing to start because it could not allocate enough memory.

I was going to suggest that you check the downloaded chunks in your torrent client but KTorrent is crashing and I cannot access it to show the procedure. It is somewhere in there.

I've checked the file length and it is "1486893056 2010-04-13 23:39 phoenixos030410.iso", which translates to 1418.0 MiB but this is allocated at the start so it does not mean that the file is fully downloaded. Also fbd2e608cf872a3ded2ff8711daa0e6b phoenixos030410.iso.

Currently I am trying to get a torrent stared again here but am having problems. Will report back when I succeed.

Paul
avatar
PGTips91

Posts : 71
Points : 75
Reputation : 0
Join date : 2010-01-28
Age : 76
Location : Auckland, New Zealand

View user profile http://picasaweb.google.com/paulgtaylor91

Back to top Go down

Re: Finding download information

Post  PGTips91 on Thu Apr 15, 2010 6:31 am

Well I think I have my torrent seeding, 100% available, and the port-forwarding sorted. I think that it was necessary to reboot the modem for the changes to be implemented. I wonder if your torrent client can complete and verify the download now?

Paul
avatar
PGTips91

Posts : 71
Points : 75
Reputation : 0
Join date : 2010-01-28
Age : 76
Location : Auckland, New Zealand

View user profile http://picasaweb.google.com/paulgtaylor91

Back to top Go down

Re: Finding download information

Post  StanTheMan on Thu Apr 15, 2010 7:30 pm

I am starting from scratch.
My torreent client running on your URL ,
but it reports

Downloading from 0 of 0 connected peers - idle .

StanTheMan

Posts : 79
Points : 81
Reputation : 0
Join date : 2010-01-29
Age : 97
Location : Kamloops BC Canada

View user profile

Back to top Go down

Re: Finding download information

Post  StanTheMan on Thu Apr 15, 2010 8:02 pm

My torrent has now started downloading

Downlooading from 1 of 1 connected peer - Down: 7 KB/s

StanTheMan

Posts : 79
Points : 81
Reputation : 0
Join date : 2010-01-29
Age : 97
Location : Kamloops BC Canada

View user profile

Back to top Go down

Re: Finding download information

Post  PGTips91 on Fri Apr 16, 2010 12:09 am

StanTheMan wrote:My torrent has now started downloading

Downloading from 1 of 1 connected peer - Down: 7 KB/s

My torrent 'health' still does not show green, as it should, and has a red indicator saying that it is blocked by a firewall.

I ain't sure what's up with it. Done my best to set the modem right, rebooted it and everything else is fine. I find this sort of work rather frustrating as I cannot remember which options I have tried or what I've just done seconds ago, so keep going around in circles.

Let me know if you get a good download eventually. I'll keep trying things at my end. Hopefully I will find out how to do this right someday!

Paul
avatar
PGTips91

Posts : 71
Points : 75
Reputation : 0
Join date : 2010-01-28
Age : 76
Location : Auckland, New Zealand

View user profile http://picasaweb.google.com/paulgtaylor91

Back to top Go down

Re: Finding download information

Post  Deffi on Fri Apr 16, 2010 2:18 am

As I had already download phoenix when you first posted the aria2 method and as posted, all appeared OK. Last evening (UK) I tried to find your torrent but could not. There is one on pirate bay which has not shown up before. I tried the Aria2 + and was told I had already downloaded. The aria command line worked but the torrent line did not connect.

David
avatar
Deffi

Posts : 106
Points : 126
Reputation : 0
Join date : 2010-01-30
Age : 76
Location : UK

View user profile

Back to top Go down

Re: Finding download information

Post  StanTheMan on Fri Apr 16, 2010 11:12 am

My torrent has been running now for 15 hours. For a long time it was idle , but now has started downloading at about 100 KB/s and has downloaded 19 $ of your ISO.
Will keep it running.

StanTheMan

Posts : 79
Points : 81
Reputation : 0
Join date : 2010-01-29
Age : 97
Location : Kamloops BC Canada

View user profile

Back to top Go down

Re: Finding download information

Post  StanTheMan on Fri Apr 16, 2010 12:51 pm

After some lenghy idle periods , my download is now just past 50 % running at about 100 Kb/s.
Shows downloading from 1 of 1 connected peers.
If someone else wants to get onto this torrent , here is the URL

http://pgtips91.pbworks.com/f/phoenixos030410.iso.torrent

Make sure you set the target location to your target folder to save.

StanTheMan

Posts : 79
Points : 81
Reputation : 0
Join date : 2010-01-29
Age : 97
Location : Kamloops BC Canada

View user profile

Back to top Go down

Re: Finding download information

Post  StanTheMan on Fri Apr 16, 2010 2:50 pm

My ISO download has just completed.
The last two or three hours the download rate was steady at about 100 KB/s.
I will leave it on the seeding mode for a bit.
Later will burn the install disk and see if it installs.

StanTheMan

Posts : 79
Points : 81
Reputation : 0
Join date : 2010-01-29
Age : 97
Location : Kamloops BC Canada

View user profile

Back to top Go down

Re: Finding download information

Post  StanTheMan on Fri Apr 16, 2010 7:56 pm

I have burned the ISO
It loads Live Phoenix okay ,
but when I type ' install ' at the boot prompt , it just loads Live.
No apparent way to start installer.

StanTheMan

Posts : 79
Points : 81
Reputation : 0
Join date : 2010-01-29
Age : 97
Location : Kamloops BC Canada

View user profile

Back to top Go down

Re: Finding download information

Post  PGTips91 on Fri Apr 16, 2010 9:06 pm

StanTheMan wrote:I have burned the ISO
It loads Live Phoenix okay ,
but when I type ' install ' at the boot prompt , it just loads Live.
No apparent way to start installer.

Hi Stan,

You have to go right back to the first post on the release to find the answer.

Flamehawk Release

Bug #2: Installer: on the desktop in Live mode, right click your mouse and pick run command and type ubiquity. The installer will start fine.

That should do it.

Paul
avatar
PGTips91

Posts : 71
Points : 75
Reputation : 0
Join date : 2010-01-28
Age : 76
Location : Auckland, New Zealand

View user profile http://picasaweb.google.com/paulgtaylor91

Back to top Go down

Re: Finding download information

Post  Sponsored content


Sponsored content


Back to top Go down

Page 1 of 2 1, 2  Next

View previous topic View next topic Back to top

- Similar topics

 
Permissions in this forum:
You cannot reply to topics in this forum