CODEWITHSUNDEEP
asynchronousf#
M.Y. Babt
M.Y. Babt

Reputation: 2891

F#: Downloading data asynchronously

I am new to programming and F# is my first language.

Here are the relevant parts of my code:

open System.IO
open System.Net

let downloadHtmlFromUrlAsync (url: string) =
    async { 
        let uri = new System.Uri(url)
        let webClient = new WebClient()
        let! html = webClient.AsyncDownloadString(uri)
        return html
        }

let downloadHtmlToDisk (url: string) (directoryPath: string) = 
    if isValidUrl url then
        let name = getNameFromRedirectedUrl url
        let id = getIdFromUrl url
        let html = downloadHtmlFromUrlAsync url
        let newTextFile = File.Create(directoryPath + "\\" + id.ToString("00000") + " " + name.TrimEnd([|' '|]) + ".html")
        use file = new StreamWriter(newTextFile) 
        file.Write(html) 
        file.Close()

let downloadEntireDatabase (baseUrl: string) (totalNumberOfPeople: int) = 
    let allIds = [ for i in 1 .. totalNumberOfPeople -> i ]

    allIds
    |> Seq.map (fun id -> baseUrl + string(id))
    |> Seq.filter isValidUrl
    |> Seq.map downloadHtmlToDisk
    |> Async.Parallel 
    |> Async.RunSynchronously

I have tested the functions isValidUrl, getNameFromRedirectedUrl, getIdFromUrl in F# interactive. They work fine.

My problem is this: When I try to run the code pasted above, the following error message is produced:

Program.fs(483,8): error FS0193: Type constraint mismatch. The type seq<(string -> unit)> is not compatible with type seq<Async<'a>> The type Async<'a> does not match the type string -> unit

What went wrong? What changes should I make?

Upvotes: 2

Views: 141

Answers (1)

Random Dev
Random Dev

Reputation: 52280

The problem is probably this line (can you please give us the definition of downloadFighterHtmlToDisk):

  allIds
    ...
    |> Seq.map downloadFighterHtmlToDisk
    ...

based on the error message this functions seems to have a signature string -> string -> unit but you really need string -> Async<'something>.

Now I guess you used downloadHtmlToDisk or something similar and you can but then I would suggest rewriting it to:

let downloadHtmlToDisk (directoryPath: string) (url: string) = 
    async {
        if isValidUrl url then
            let name = getNameFromRedirectedUrl url
            let id = getIdFromUrl url
            let! html = downloadHtmlFromUrlAsync url
            let newTextFile = File.Create(directoryPath + "\\" + id.ToString("00000") + " " + name.TrimEnd([|' '|]) + ".html")
            use file = new StreamWriter(newTextFile) 
            file.Write(html) 
    }

and use it like

 let downloadEntireDatabase (baseUrl: string) (totalNumberOfPeople: int) = 
        let allIds = [ for i in 1 .. totalNumberOfPeople -> i ]

        allIds
        |> Seq.map (fun id -> (id, baseUrl + string(id)))
        |> Seq.filter (fun (_,url) -> isValidUrl url)
        |> Seq.map (fun (id,url) -> downloadHtmlToDisk (getFighterPath id) url)
        |> Async.Parallel 
        |> Async.RunSynchronously

See the let! html = ..? This is important - this is where the async will happen ;) - if you want you can find similar operations to write your file asynchronously. Also you don't need to close your file - dispose should handle it

remark

I have just seen that you reextract the id from the url - you might also use this instead of the way I used tuples but I think it's better to really pass the id on if you still need it - for example in downloadHtmlToDisk you really need the id and could have created the url from the id there instead - a much easier approach IMO but I don't want to rewrite everything you go - just experiment a bit with this stuff

Upvotes: 2

Related Questions