- LFTP Scripts
If you’re not familiar with, LFTP it bills itself as a “sophisticated file transfer program”. It supports common protocols like FTP, HTTP/s, FXP, SFTP, and BitTorrent. It can also download multiple files in parallel, download one file as multiple segments in parallel, and resume a paused or canceled download. These capabilities make it particularly attractive to users of seedboxes, as the servers are generally geographically far from the user, with limited and or spotty connectivity.
If you’re on Linux, you should find LFTP in your distribution’s package repository. It’s also easily available on OS X via Homebrew. If neither of those work for you, the LFTP site contains source code and rpm binaries. If you’re on Windows, it appears to be readily available in Cygwin.
# Ubuntu / Debian $ sudo apt install lftp # Centos / RHEL / Fedora $ sudo yum install lftp # Mac OS X via Homebrew $ brew install lftp # Windows via Cygwin # ( untested, please let me know if it works, firstname.lastname@example.org ) > setup-x86_64.exe -q -N -n -d -D -s https://mirrors.sonic.net/cygwin -P lftp
LFTP consists of a number of commands. LFTP will give you an interactive shell to call these commands, like an FTP or shell session. You can also pass commands directly to LFTP with no interactivity, more like rsync. Each of LFTP’s commands does different things and takes different options. Some can be strung together to do great things, but unless you use LFTP frequently you’ll never remember them. That’s why I wrote down these common example use cases so you can get going quickly:
Note: I’ve seen some other blogs about LFTP that show how to setup an FTP server so you can download your files with LFTP. If you have SSH access to the remote server, just use SFTP over SSH and save yourself the hassle and diminished security of running an FTP server.
- Connect to a server via SSH, get dropped into an LFTP shell. LFTP will use SFTP over SSH so you don’t have to setup another server:
- Connect to an FTP server with authentication, get dropped into an LFTP shell:
lftp -p21 username ftp://ftp.ucsb.edu
- Connect to an HTTP/S server, get dropped into an LFTP shell.:
Once you’re connected to the remote server, you can start transferring files:
- Download a file with pget with continue support in 8 segments:
lftp> pget -c -n 8 ./debian-6/debian-6.i386.iso
- Mirror a directory using pget with continue support, 4 concurrent files and 8 concurrent segments per file:
lftp> mirror -c -P 4 --use-pget-n=8 ./debian-6
- Download multiple files with a glob expression, continue, pget, and 12 segments:
lftp> glob -- pget -c -n 12 ./debian-6/*
- Mirror multiple directories slected by a glob, using continue, pget for 12 segments, and mirror for 4 concurrent files:
lftp> glob -d -- mirror -c -P 4 --use-pget-n=12 ./debian*
- Download a torrent with lftp.
$ lftp lftp :~> torrent ./ubuntu-21.04-desktop-amd64.iso.torrent Name: ubuntu-21.04-desktop-amd64.iso dn:2.9G up:0 complete, ratio:0.00/0.00/0.00 peers:17 connected:0 active:0 complete:0 Seeding in background..
- Upload a directory by reversing the
lftp> mirror -c --reverse ./remote_target ./local_source
- Update a remote directory with
mirror, removing (from the remote side) anything that’s been deleted locally:
lftp> mirror -c --delete --reverse ./remote_target ./local_source
- Upload a set of files based on a glob, with continue support, and 4 concurrent files. Note, if you’re transferring multiple files in a directory hierarchy,
mirroris much easier to work with:
lftp> mput -c -P4 ./*.mp4
- Execute some LFTP commands and keep the shell open with
lftp -e "cd A; pget A.txt; cd ../B; pget B.txt" ftp://ftp.ucsb.edu lftp>
- Execute an LFTP script file with
lftp -f ./script.lftp
Let’s look at a few other common use cases we can solve with an LFTP script:
- Our LFTP script downloads
local_dirwith mirror and pget (2 files, 4 segments, continue). This will not remove local files that are missing on the remote directory, so you can use this to accumulate files that live on your seedbox ephemerally:
# script.lftp open sftp://10.0.0.1 mirror -c -P 2 --use-pget-n=4 ./remote_dir /mnt/local_dir quit
- If you want to keep the local directory synced with the remote one, you can add the
--deleteflag to delete files locally when they’re deleted on the remote end. This is more useful if you’re using mirror to sync something like a website directory:
# sync.lftp open sftp://10.0.0.1 mirror --delete -c -P 2 --use-pget-n=4 ./remote_dir /mnt/local_dir quit
- If you lose your server or need to restore your website from an LFTP sync backup, you can use the
mirrorto upload the files instead of downloading them. Note that using
pgetwon’t make a difference here:
# resore.lftp open sftp://10.0.0.1 mirror --reverse --delete -c ./remote_dir /mnt/local_dir quit
I originally wrote most of this article as a sort of “cheat sheet” while I was working in an IT shop at UCSB. I wanted to mirror it here as I no longer work in that shop and would be bummed to lose the article. You can find it here if you want.