Saturday, September 20, 2008

(wget = DM) != GUI

Wget the default download manager in *nix systems is a very versatile tool. I will be focusing on some of its very important features useful to a end user. So here we begin :

  1. Simply downloading a file :
    $ wget <URL>
    Or better :
    $ wget -c <URL>
    -c or --continue lets you resume your incomplete download done using wget or a different program

  2. Downloading a file silently (a very imp. feature in wget, to work in background) :
    $ wget -q <URL> &
  3. Taking a log of all messages :
    $ wget <URL> -o filename
  4. Downloading files from a Queue :
    $ wget -i filename
    Strore all download links in a file separated by Return key and link it with wget
  5. Set your no. of retries and timeout seconds :
    $ wget -t <no. to tries> -T <duration> <URL>
  6. Displaying the content of a file rather than saving it on Hard Disk :
    $ wget -O - <URL>
  7. Retrieving all file with a particular extension from a web directory :
    $ wget -r -l1 -np -A.<extension> <directory>
    Example for http://www.abc.com/def/*.jpg
    $ wget -r -l1 -np -A.jpg http://www.abd.com/def/
  8. Downloading an entire Web Site :
    $ wget -c -r -nc -p -D <domain> -k <URL>
    For this I have written a script for easy downloading :
    #!/bin/bash

    # Bash Script for Web Site downloading by GNUger
    # Syntax : $ sitesdwn [URL]

    command="wget -c -r -p -k"
    domain=$(echo $1 | cut -d/ -f3 | cut --complement -d. -f1)

    echo
    read -p "Download within $domain only (y/n) : " opinion
    if [[ $opinion == 'y' ]] ; then
    command="$command -D $domain"
    fi
    read -p "Disable overwriting existing files (y/n) : " opinion
    if [[ $opinion == 'y' ]] ; then
    command="$command -nc"
    fi
    read -p "Download outside the current directory (y/n) : " opinion
    if [[ $opinion == 'n' ]] ; then
    command="$command -np"
    fi
    read -p "Save as HTML extension (y/n) : " opinion
    if [[ $opinion == 'y' ]] ; then
    command="$command -E"
    fi
    read -p "Enable compatibility with Windows (y/n) : " opinion
    if [[ $opinion == 'y' ]] ; then
    command="$command --restrict-file-names=windows"
    fi
    echo "Downloading ... "
    echo
    $command $1
    echo
    echo "End"
    exit 1
    fi
    Or download the script from here. Syntax :
    ./sitedwn.sh <URL>


This is not it.. there's hell lot more but beyond the scope of this post. So just read the Manuals (worth reading specially the Examples part). There are GUI version for wget :
  • GWget - For Gnome
  • Kget - For KDE
But they does not provide you so many options and manual customizations as wget (CLI) does.

References : man pages of wget, Linux Journal

9 comments:

- said...

Nice less Geeky blog you have going!!!
I invite you to visit my blog @ blabweb.com and enter yourself in a contest to win a blog review by me! You have to subscribe to my RSS feed by email and let me know you have entered the contest.

alx said...

hey cool blog....really useful 1....though i stil use windows due 2 my knowledge in linux(or lack f it)...wil cum bak 4 more

downlinks said...

Thnx for dropping comments.

you are also doing good job, i always love to learn about linux, in past i also tried some distro like readhat and ubuntu, i m planing to deploy mandriva on my system but due to lack of time unable to complete my wish.
I hope you will keep visiting my blog
http://tech-blog.iblogger.org/
if u will like i can give you author rights, so u can right linux posts on my blog.

Unknown said...

hey real nice one buddy ...

I started linux only recently .. iam amazed by the flexibilities linux can offer ..

ur blog was an eye opener for me :)

Anant Khaitan said...

I thank everyone for their comments, looking forward for more of participation from your side..

@ orkut: Thanks for the invitation.. will surely inform you

@ kartik: thanks for that eyeopener complement :)

Unknown said...

Nice blog; most of the contents/posts are interesting.

This is my unix bash scripting blog with awk and sed posts, thought of sharing with you.

//Jadu, unstableme.blogspot.com

Anant Khaitan said...

@Jadu : Thanks for the share

jmsajid said...

i need jil 0606 tv card driver plz help
jmsajid@gmail.com

Unknown said...

Wget is indeed one seriously awesome program.