Read folder paths from FTP directory to IEnumerable

Question

I'm currently working on a .NET 4.6 console application. I need to parse a couple of XML files from different directories on my FTP server. I thought the best approach would be, to read all file paths and store them into an IEnumerable, to process them further (Serialize XML Files to objects).

The root FTP path looks like this:

string urlFtpServer = @"ftp://128.0.1.70";

File paths look like this:

string file1 = @"ftp://128.0.1.70/MyFolder1/Mainfile.xml";
string file2 = @"ftp://128.0.1.70/MyFolder1/Subfile.xml";
string file3 = @"ftp://128.0.1.70/MyFolder2/Mainfile.xml";
string file4 = @"ftp://128.0.1.70/MyFolder2/Subfile.xml";
string file5 = @"ftp://128.0.1.70/MyFolder3/Mainfile.xml";

My question is, do you know how I can get those specific file paths?

I currently can read the folders of my FTP directory with this coding:

static void Main(string[] args)
{
    string url = @"ftp://128.0.1.70";

    FtpWebRequest request = (FtpWebRequest)WebRequest.Create(url);
    request.Method = WebRequestMethods.Ftp.ListDirectoryDetails;

    request.Credentials = new NetworkCredential("My-User", "mypassword");

    FtpWebResponse response = (FtpWebResponse)request.GetResponse();

    Stream responseStream = response.GetResponseStream();
    StreamReader reader = new StreamReader(responseStream);
    Console.WriteLine(reader.ReadToEnd());

    Console.WriteLine("Directory List Complete, status {0}", response.StatusDescription);

    reader.Close();
    response.Close();

    Console.ReadKey();
}

Do you know how I can read all file paths from the FTP main directory and possibly store them into a List<string>?

Thank you very much!!

Answer 1

Using FtpWebRequest

The FtpWebRequest does not have any explicit support for recursive file operations (including listing). You have to implement the recursion yourself:

• List the remote directory
• Iterate the entries, recursing into subdirectories (listing them again, etc.)

Tricky part is to identify files from subdirectories. There's no way to do that in a portable way with the FtpWebRequest. The FtpWebRequest unfortunately does not support the MLSD command, which is the only portable way to retrieve directory listing with file attributes in FTP protocol. See also Checking if object on FTP server is file or directory.

Your options are:

• Do an operation on a file name that is certain to fail for file and succeeds for directories (or vice versa). I.e. you can try to download the "name". If that succeeds, it's a file, if that fails, it's a directory.
• You may be lucky and in your specific case, you can tell a file from a directory by a file name (i.e. all your files have an extension, while subdirectories do not)
• You use a long directory listing (LIST command = ListDirectoryDetails method) and try to parse a server-specific listing. Many FTP servers use *nix-style listing, where you identify a directory by the d at the very beginning of the entry. But many servers use a different format. The following example uses this approach (assuming the *nix format)

void ListFtpDirectory(
    string url, string rootPath, NetworkCredential credentials, List<string> list)
{
    FtpWebRequest listRequest = (FtpWebRequest)WebRequest.Create(url + rootPath);
    listRequest.Method = WebRequestMethods.Ftp.ListDirectoryDetails;
    listRequest.Credentials = credentials;

    List<string> lines = new List<string>();

    using (FtpWebResponse listResponse = (FtpWebResponse)listRequest.GetResponse())
    using (Stream listStream = listResponse.GetResponseStream())
    using (StreamReader listReader = new StreamReader(listStream))
    {
        while (!listReader.EndOfStream)
        {
            lines.Add(listReader.ReadLine());
        }
    }

    foreach (string line in lines)
    {
        string[] tokens =
            line.Split(new[] { ' ' }, 9, StringSplitOptions.RemoveEmptyEntries);
        string name = tokens[8];
        string permissions = tokens[0];

        string filePath = rootPath + name;

        if (permissions[0] == 'd')
        {
            ListFtpDirectory(url, filePath + "/", credentials, list);
        }
        else
        {
            list.Add(filePath);
        }
    }
}

Use the function like:

List<string> list = new List<string>();
NetworkCredential credentials = new NetworkCredential("user", "mypassword");
string url = "ftp://ftp.example.com/";
ListFtpDirectory(url, "", credentials, list);

---

Using 3rd party library

If you want to avoid troubles with parsing the server-specific directory listing formats, use a 3rd party library that supports the MLSD command and/or parsing various LIST listing formats; and recursive downloads.

For example with WinSCP .NET assembly you can list whole directory with a single call to the Session.EnumerateRemoteFiles:

// Setup session options
SessionOptions sessionOptions = new SessionOptions
{
    Protocol = Protocol.Ftp,
    HostName = "ftp.example.com",
    UserName = "user",
    Password = "mypassword",
};

using (Session session = new Session())
{
    // Connect
    session.Open(sessionOptions);

    // List files
    IEnumerable<string> list =
        session.EnumerateRemoteFiles("/", null, EnumerationOptions.AllDirectories).
        Select(fileInfo => fileInfo.FullName);
}

Internally, WinSCP uses the MLSD command, if supported by the server. If not, it uses the LIST command and supports dozens of different listing formats.

(I'm the author of WinSCP)