Globally fast – a critical USP for the secure file sharing service FTP-Stream. But why are some links slow in the first place and what are the techniques for accelerating file transfer? If you’ve tried sending large files to China you’ll be familiar with excruciatingly slow file transfer speeds, timeouts and failed transfers, even though both the the sending and receiving parties are on fast links to the Internet. So what’s the underlying problem and what are the available solutions.

Well the heart of the problem is in Transmission Control Protocol (TCP)  Common Internet applications such as the Web, Email and FTP rely on TCP  for controlling the flow of data across the local network and the Internet.

TCP – a quick look at the culprit

When data is transferred over the Internet it’s the job of IP (that’s Internet Protocol) to send packets, but without a control mechanism it’s not reliable. TCP was conceived in 1974 to control the flow of packets and is optimised for accurate delivery rather than fast delivery.  TCP guarantees that all bytes are received and in the correct order.  The sender keeps a record of each packet sent and expects to receive a timely acknowledgement, and this is where the problem arises.

Over long or high loss links the acknowledgment can take so long to arrive, the sender assumes the packet is lost and stops sending to retransmit the ‘lost’ packet, this accounts for the jerky stop-start file transfers we’ve all seen over long or flaky links.

WAN optimization

Of course there’s stuff you can do to mitigate these effects. See this Wikipedia article for the tech details on WAN optimization, but for a hosted FTP service like FTP-Stream the options are limited as we only control one end of the link.

UDP based solutions

The User Datagram Protocol (UDP) was designed in 1980 and unlike TCP does not require any dialogue between the sender and receiver. The source just pumps packets at the target in a stream, there’s no control – that’s great if you’re streaming a movie where the overwhelming need is to maintain the flow and the odd lost packet doesn’t much matter, but for most file transfer uncontrolled UDP transfer are clearly unsuitable.

However products like Aspera  exploit the faster transfer potential of UDP by replacing TCP with a proprietary control mechanism. It does work – the problem however  is that it’s a hardware based and you need the hardware at both ends of the link. So that’s OK for static connections between two parties but is not suitable for cloud large file transfer like FTP-Stream where most users will not have the specialist hardware.

The FTP-Stream Solution

As a managed file transfer solution with millions of user worldwide we need to support standard access methods such as FTP, SFTP, and secure Web which all rely on TCP. So FTP-Stream uses two techniques: multi-path and parallelization. Our proprietary systems will break a large file transfer into small chunks and send them in parallel over SFTP for security. Although the transfer speed of each chunk is still slow, but sending man in parallel we get huge increases in speed.

Furthermore we exploit our global network and intelligent routing software to probe for faster, less congested routes. The receiving end of the link assembles the constituent parts and performs an integrity check.

The results are dramatic – for example a 15 gig file transfer from New York to a south-east Asia destination, which would take days over the public Internet reaches its destination in just two-three hours

If you have any comments or questions about large file transfer over high latency links you can mail me antony@maytech.net, we like nothing better than talking about large file sharing.

I launched FTP-Stream as a hosted FTP service in 2005 thinking it would be useful in specific and obvious sectors such as print and publishing, creative agencies, film and TV. Over the years since I have been amazed by the diversity of the customer base, that’s diversity by geography, industry sector and business size. The range could not be greater.

For example there’s the marketing department of any medium to large company who needs to share large image or video files with press and suppliers; the oil and gas sector transfer huge file – seismic data for analysis for example. And mostly they come to FTP-Stream as we have no file size limits. Same goes for tech businesses who need to share huge disk images worldwide – files are often 50 gig!

FTP-Stream is used for document control on large and complex projects such as the development of the 2012 Olympic facilites as well as the running of the London games.

The composer Rossini once said “Give me a laundry list and I will set it to music”, I’d like to challenge him with our list of customer sectors:

Academia, Air Traffic Control, Animators, Architects, Automotive, Aviation, Biotechnology, Brewers, Broadcasters, Charities & NGOs, Chemicals, Commodity Traders, Construction, Creative agencies, Data Capture, Defense, Direct Mail, Education, Electronics, Energy, Engineering, Entertainment, Exhibition & Display, Fashion, Film & TV, Food Industry, Formula 1, Gaming, Government Agencies, Insurance, Investment Banking, IT Support, Law, Litigation Support, Manufacturing, Maritime, Metals & Minerals, Medical, Metrology, Music, Oil & Gas, Olympics 2012, Packaging, Pharmaceuticals, Photographers, Photo Libraries, Political Parties, Printers, Post Production, Public Relations, Publishers, Real Estate, Retail, Software, Steel, Theatre, Theatrical Agencies, Translators, Travel, Trucking, Venture Capital, Zoological Gardens.

Geographical distribution is also interesting, we have customers in 40 countries and users everywhere. We recently got our first customer in Mongolia – seems they’ve hit pay dirt there!

Of course not all file transfer is interactive – FTP-Stream is widely used for machine to machine automated file transfer particularly using FTP, SFTP and FTPS. There’s a few anecdotes that illustrate that:

When you’re checking  the electronic menu board in a leading US burger chain think how the data on the display is fed and refreshed …

You’re driving on the highway on a newly built embankment, probably not too worried about landslips? That’s OK, because motion detectors will be embedded and instantly reporting any movement to a control center …

Still on the highway and tuned into local radio, did you wonder how a small local station is fed their content: music, international news, weather …

Hopefully this doesn’t happen but say you need medical transportation  - how are the despatch orders managed and distributed by the largest provider in the Southern US…

Congratulations you guessed, all through FTP-Stream. So whether you’re watching a movie, eating in a restaurant, going to the Zoo, going to the races, watching the Olympics… or even, in some countries, paying your taxes…  OK I won’t go on, but you get the point.

If you have any comments or questions about secure file transfer you can mail me antony@maytech.net, we like nothing better than talking bizarre file sharing workflows.

FTP-Stream is a cloud managed file transfer service offering customers a number of access methods (protocols) including the three encrypted protocols: FTPS, SFTP and HTTPS.

HTTPS is of course secure web access (HTTP over SSL),  familiar as the method providing security when accessing e-commerce websites for example.

SFTP and FTPS are both secure file transfer protocols supported by most FTP client software such as Filezilla or CuteFTP on the PC and Cyberduck or Transmit on the Mac.

The first thing you need to know is that security / encryption strength is not an issue as they use similar encryption algorithms. So what are the differences?

FTPS is FTP over SSL – much like a secure website is HTTP over SSL. The client initiates the connection to the server to setup the encrypted SSL session. On the server side the X.509 certificate includes the public key and verifiable information about the certificate owner, more on this below…

Next the client opens a standard FTP session over the encrypted SSL session. FTP uses two channels on ports 21 (the control channel) and a randomly negotiated port for the data channel and this is where FTPS can run into firewall problems. Since the session is encrypted the firewall does not understand the conversation and therefore does not open the randomly selected port for the data channel.

SFTP stands for SSH File Transfer Protocol – a set of FTP-like commands for file transfer as an extension of the SSH (secure shell) protocol.

SFTP runs on a single port (22) and that means that firewall problems are rare. SFTP is a secure encrypted protocol but has no mechanism for public authentication of the server identity.

Keys and Certificates

Under normal interactive operation there is no need to manually generate key pairs for SFTP or FTPS, the client software and the server handle this automatically.

However Maytech is able to install an SSL certificate for identity assurance under FTPS. So just like a secure website the user can be assured of the identity of the server using the public key infrastructure with certificates signed by a certificate authority such as Verisign. This means that users can connect to a host such as ftp.mydomain.com and be assured of its identity.

Automated File Transfer

Both protocols are widely used for secure machine to machine file transfer. There are SFTP and FTPS components in many common programming languages.

FTP-Stream users can optionally generate an SFTP key pair and upload the public key to the server and that’s a common scenario for automated file transfer as it makes the scripting easier.

Which is the best protocol for my application

Generally we recommend SFTP as you are less likely to encounter firewall problems. But choose FTPS where you wish to rely on the public key infrastructure to assure identity.

If you are setting up automated machine to machine file transfer you’ll need to consider availability of software components, for example there’s built-in support for FTPS in .NET but you’ll have to rely on third party components if you wish to use SFTP.

If you have an questions about secure file transfer you can mail me antony@maytech.net, we’d love to hear from you.

Current FTP-Stream customers will know that we are working hard on FTP-Stream version 3. Actually more than an update, it’s a complete ground up rewrite a fresh streamlined management and file transfer interface.

Responding to customer requests and building on our core USPs, there’s plenty of new functionality enhancing: security, accountability, branding and ownership, global acceleration and support for file transfer protocols beyond the limits web browser such as FTP and SFTP.

Multiple Global Access Points

FTP-Stream is available at New York, Hong Kong and London. We’re now building out our points of presence and will add around a dozen in 2012. The first two in Mainland China are already up and running,  and will soon be followed by: Dubai, Sydney, US West Coast and Tokyo. Each will have accelerated connectivity to our core network enhancing our “always local, always fast” offer.

FTP(S) and SFTP Support

Whereas many competitive services work over the Web only, our strength is support for the three major file transfer methods (HTTP, SFTP and FTP including FTPS). However so far some of advanced functions like folder sharing have been restricted to the Web interface. Folder sharing and permissions will be hugely flexible in V3 with identical functionality across all protocols.

Security and Access Controls

Two frequent requests will be included – restrictions by IP and two factor authentication.

Previews

V3 will offer previews of all ‘viewable’ file types, that includes all image formats, Office documents, PDFs, and audio files.

File and Folder History and Search

Comprehensive logging and reporting is vital for accountability and compliance. This will be greatly enhanced in V3 including instantly available history for every file and folder. Furthermore, using this data users will be able to filter file and and folder listings using a wide range of criteria including user applied tags.

Open API

We’ll release an open API for developers and system integrators to build managed file transfer into third party applications.

Management Interface

At last we’ll combine the administration panel with Web file management, and of course introduce a long list of usability improvements.

Desktop App

A comprehensive desktop app for Mac and Windows providing full management and file transfer functionality.

Schedule

We’ve frozen development on the current version to concentrate on V3 – tentatively scheduled for release late July 2012.

If there’s something on your wishlist please mail me antony@maytech.net, we’d love to hear from you.

Often, simple files sharing by many departments can ultimately turn into a requirement for a central policy as the need for accountability, control and enhanced security arises, so scalability is often good to consider at the outset.

With the above in mind, and presuming secure file transfer is a must, in most cases you’ll also need to consider how you want your staff and customers or partners to send and receive data – over the web, via  FTP or secure FTP or a mix of all three.

Many companies are still using FTP so flexibility in the method used should be a consideration too. Many solutions market themselves as replacements for FTP but FTP is reliable, will be around for a long time to come and is also secure if you use SFTP. Flexibility in access method is important.

Here are some file transfer services to consider.

On Premise solutions
Microsoft Sharepoint – a sophisticated online collaboration tool but mostly used within the enterprise due to security considerations.

In-house FTP server software such as IPSwitch or CuteFTP server for installation on corporate servers providing controlled external access to files and folders on the LAN. This is generally a license model and makes use of local internet up-links and require ongoing server and firewall administration.

Dedicated file transfer solutions such as Aspera and FileCatalyst offer proprietary managed file transfer. By exploiting the UDP protocol such solutions claim faster transfer speeds especially over long links. Dedicated client software is needed on every point to transfer files.

Cloud solutions
Instant or casual, consumer origin file sharing – two well known examples Dropbox and YouSendIt, offer low initial costs and little investment in setup and are easy to use. They are turning their attention to business use in recent time. Files are transferred on a ubiquitous platform which is a consideration where corporate identity is an important factor and they often require users to register in order to download files. There can also be limits to file sizes too.

Solutions geared towards business to business file exchange include Sharefile, FTP-Stream or Egnyte. All have relevant business features. Some offer good branding potential with FTP-Stream offering complete ownership of your identity such that it is often perceived as an extension to the corporate website.

Also Box.net – now Box.com is a sophisticated service but principally tuned to online collaboration as opposed to heavy duty external file transfer. If your business is global ensure that the service has a global network and check the file size limits as well as the access method it supports.

We’d love to hear any comments or experiences on the services outlined in this post and if you would like a free consultation about your file transfer requirements get in touch with us us here.

FTP service providers trumpet expressions such as ‘AES-256 at rest encryption’, sounds impressive but what does it mean in practice and what’s the true value of on-disk encryption?

OK so you’re carrying around a USB stick with the nation’s secrets or FedExing a drive loaded with healthcare records – two scenarios where encryption is both effective and vital. The decrypt key is recorded or communicated separately from the encrypted media which if compromised yields unreadable data.

Take another scenario – a cloud file sharing provider boasts on-disk encryption, sounds good, but bear in mind that before files are downloaded they must be decrypted, to decrypt you need the key, the key will be stored proximate to the encrypted drive. Which of course means that a hacker who has penetrated the system, or a malicious insider can read files just as if they were never encrypted.

For this exact reason, according to Information Week, popular online storage provider Dropbox, who proudly proclaim AES-256 encryption, was forced to modify its claim from:

“Dropbox employees aren’t able to access user files…”

and now:

“we have strict access controls that prohibit employee access to user data.”

So where does Maytech stand on this with its FTP and SFTP hosting service? Well firstly we believe that security is more about continuous examination of all our facilities, procedures and code, and since retired media never leaves our tier 3 data centers alive, we say that on-disk encryption adds little significant benefit. However, this technology has entered several public standards and is increasingly requested by customers — sometimes you have to go with the flow… So yes OK we’ll tick that box and will be rolling out on-disk encryption for all our services in Q1 2012.

Much more interesting and relevant to secure cloud file sharing is end-to-end encryption, and I’ll come to that topic in a future blog

If you think technology can solve your security problems, then you don’t understand the problems and you don’t understand the technology. — Bruce Schneier

Latency and congestion at the international gateway leads to slow data rates and broken transfers. However if you’re a business running on a tight schedule you need a reliable and fast method of electronic file exchange to ensure that you are in control of the delivery of your files.