"Accelerated" file transfer software over poor WANs, e.g. via UDP

[occamsrazor]

I regularly have to send image and video files, ranging from 1MB to 200MB half way around the world, and currently we use FTP. Files are sent over a variety of internet connections including internet cafes, 3G and satellite. Sometimes the connections are congested (e.g. overloaded cell towers) and sometimes the latency is very high indeed (e.g. 1000-2000ms over satellite). Sometimes I also suspect weird traffic-shaping is also going on. Other times the ISP network infrastructure is just badly planned or implemented. The key aspect is that I DO NOT have control over any of these networks.

So I have been looking for a better system of file transfer, and in doing so I've come across various options for transferring files over UDP instead of TCP. Which may or may not be the answer - I don't know - but I suspect FTP over TCP does not obtain optimal speed in such conditions. Perhaps I just need something that breaks files up and sends over multiple sessions/connections simultaneously, I also don't know.

It has to be able to traverse firewalls, achieve optimal speeds for any given connection (and work over a variety of such connections), and require no router network setup at each end - i.e. work from anywhere to anywhere - ideally. It can be server-client based with different software for server and client, or it can be peer-to-peer. It does not need to be "friendly" to other users on a shared connection - in fact I want it to suck up all the available bandwidth if possible. Platform-wise it needs to work on Mac OSX at the client end, at the server end either Mac OSX or Windows. Anything else is a bonus.

I'm aware of commercial options that claim to achieve improved file transfer speed, such as:

Aspera
FileCatalyst
Expedat

But I am more interested in any open-source/free type offerings. A few I have been reading up on include the following protocols:

UFTP
Tsunami UDP
UDT

Does anyone have experience in such matters or have knowledge of improving file transfer speed over networks with high-latency and other problems?

My other thought was to somehow leverage the Bittorrent protocol which from my experience is extremely robust and also about the only thing capable of really maxxing out my network connection. But I can't see a user-friendly way to automate the transfer process when sending many small files sequentially (e.g. the image files), and the whole "swarm" aspect of it is un-needed in my case as the transfer just needs to go point-to-point.

Any thoughts?

 

[coxhaus]

I don’t know the best way to transmit data today and what tools to use but TCP was designed in the olden days when networks had errors. TCP does file integrity checks so files do not become corrupted on the network. UDP is a blast all and hope it arrives safely and in order. UDP is really better with high quality networks and video. If you get a small glitch in a video stream it really does not matter. You can use UDP but you might have to retransmit some files because they were corrupted on the network during the transfer.

 

[occamsrazor]

Thanks for your reply. You're right TCP is guaranteed delivery vs UDP alone. But it's also temperamental to errors and slows down accordingly. The UDP methods described use UDP for speed, but incorporate application-level checks to ensure file integrity and when errors occur they manage the retransmission of the failed packets. From what I understand.....

 

[DougMielke]

Airship

open disclosure - I am one of the partners in AIRWare Technology, developers of Airship.

Airship is a fully automated file transport software platform for Win, Mac and Linux - cross-platform.

It is multi-threaded TCP - up to 50 concurrent threads, all data encrypted AES-256 in transit and while at rest, able to ship up to 250,000 separate files in a session with NO overhead, 1 to 1 / 1 to many over disparate networks, auto-resume from system, network, power outages with no human intervention.

We watch it move large (200GB to 600GB per session) over any type of public or private network. Long distance, high latency networks are not a problem.

Works well in cloud and remote storage repositories and retains full encryption at rest. Manage ports remotely to eliminate the need to locally manage firewalls - all securely.

Very reasonable license terms by the month, quarter or annual.

Check out www useairware com

 

[stevech]

FTP per the standard it TCP based. If both ends use TCP windowing, which all PCs and NASes do, and if both ends properly negotiate the window size, you'll not see a speed increase with some transfer method using UDP. I don't recall seeing an option in the FTP standard for UDP.

On satellite and cellular links (the latter can have round trip delays of 300mSec or more in busy hours), TCP's windowing falters. To do the best on these links, you need either a bridge device with large buffers and the TCP option for selective retransmit. There are some older X.25 go-back-n devices used on satellite links.

But terrestrial Internet, non-cellular, ordinary windowed TCP and FTP in the passive mode should be optimal, if both ends do windowing properly.

EDITS, I recall that the TCP windowing RFC (spec) has an adaptive window size- where it varies automatically according to varying delays. Most Internet wired has non-varying delay after a route is setup. But cellular varies a lot, due to poor signals and over-subscription.