Working in the industry, at one time or another, you will have to transfer files. Im sure it will be in a variety of different ways. For the most part everyone has their favorites for each situation. But I would prefer to have one utility on all servers to handle all of those situations. So, my choice has to be a bad ass.
 

Lets look at some requirements:

  • support for all of my common protocols
  • easy and logical navigation
  • parallel threads!
  • full command line usage.  

lftp has become my tool of choice. Let me prove it with some examples:

Example 1:
At some point, everyone has had to mirror a directory that was being served by Apache with directory indexing turned on. Something like http://pkgs.repoforge.org/bsc/.
 

 debian:/tmp/outgoing# lftp
 lftp :~> open http://10.100.15.10/log/dists/lenny-20120514/binary-i386/
 cd ok, cwd=/log/dists/lenny-20120514/binary-i386                              
 lftp 10.100.15.10:/log/dists/lenny-20120514/binary-i386> mirror
 Total: 1 directory, 69 files, 0 symlinks                                                                   
 New: 69 files, 0 symlinks
 180374247 bytes transferred in 136 seconds (1.27M/s)
 lftp 10.100.15.10:/log/dists/lenny-20120514/binary-i386>

 

Downloading a Debian image from a local box. But look at the protocol… http://. I’m able to treat a web page like a cli. But it does lack the depth to do anything crazy. As far as I can see, there isnt a way to be like “mirror http://10.100.15.10/log/dists/lenny-20120514/binary-i386/a*”

Other ways to solve this? Yes. I could have done a fancy curl request stripping html using links/lynx, and then wgetting the result.

 

Example 2:

You need an entire directory copied from your server, to another server(put)… But you only have SSH.

 [root@core ~]# ls -l lame
 total 300
 -rw-r--r-- 1 root root 100002 2013-01-13 05:32 a
 -rw-r--r-- 1 root root 100003 2013-01-13 05:32 b
 -rw-r--r-- 1 root root 100004 2013-01-13 05:32 c
 [root@core ~]#

Now lets go ahead and login over sftp.

 [root@core ~]# lftp sftp://root@g1.ragenetworks.com
 Password: 
 lftp root@g1.ragenetworks.com:~> mirror -R --parallel=3 lame 
 Total: 1 directory, 3 files, 0 symlinks
 New: 3 files, 0 symlinks
 300009 bytes transferred in 2 seconds (133.2K/s)
 lftp root@g1.ragenetworks.com:~>

What I did was reverse(-R) mirror the directory. In other words I put the directory from my server, to the remove box. But I also did this using parallel threads(–parallel=N).

 

Example 3:

Along with command line usage is, how scriptable is it. There are many times you need to simply back up a directory with a cron job. This time we are going to use ftp, and script our remote commands in a file.

 [root@core ~]# cat script-file                        
 open ftp://username:password@fazey.org
 mirror -R /root/local /home/james/remote
 exit
 [root@core ~]#

Now we call lftp with the “-f” flag to give it a script input.

 [root@core ~]# lftp -f script-file 
 at 80527360 (80%) 35.12M/s eta:1s [Sending data]
 ...
 [root@core ~]#

As you can see, lftp is a hell of a tool.

Related Posts