CODEWITHSUNDEEP
asynchronousf#system.net.webexception
M.Y. Babt
M.Y. Babt

Reputation: 2891

F#: Handling web exceptions

I am new to programming and F# is my first language.

Here are the relevant snippets of my code:

let downloadHtmlToDiskAsync (fighterHtmlDirectory: string) (fighterBaseUrl: string) (fighterId: int) = 
    let fighterUrl = fighterBaseUrl + fighterId.ToString()
    try 
        async {

            let! html = fetchHtmlAsync fighterUrl
            let fighterName = getFighterNameFromPage html

            let newTextFile = File.Create(fighterHtmlDirectory + "\\" + fighterId.ToString("00000") + " " + fighterName.TrimEnd([|' '|]) + ".html")
            use file = new StreamWriter(newTextFile) 
            file.Write(html) 
            file.Close()
        }
    with
        :? System.Net.WebException -> async {File.AppendAllText("G:\User\WebScraping\Invalid Urls.txt", fighterUrl + "\n")}

let downloadFighterDatabase (directoryPath: string) (fighterBaseUrl: string) (beginningFighterId: int) (endFighterId: int) =
    let allFighterIds = [for id in beginningFighterId .. endFighterId -> id]
    allFighterIds
    |> Seq.map (fun fighterId -> downloadHtmlToDiskAsync directoryPath fighterBaseUrl fighterId)
    |> Async.Parallel
    |> Async.RunSynchronously

I have tested the functions fetchHtmlAsync and getFighterNameFromPage using F# Interactive. They both work fine.

When I build and run the solution, however, I get the following error message:

An unhandled exception of type 'System.Net.WebException' occurred in FSharp.Core.dll Additional information: The remote server returned an error: (404) Not Found.

What went wrong? What changes should I make?

Upvotes: 1

Views: 88

Answers (1)

albertjan
albertjan

Reputation: 7817

Put your try with inside the async. 

let downloadHtmlToDiskAsync (fighterHtmlDirectory: string) (fighterBaseUrl: string) (fighterId: int) = 
    let fighterUrl = fighterBaseUrl + fighterId.ToString()
    async {
        try
            let! html = fetchHtmlAsync fighterUrl
            let fighterName = getFighterNameFromPage html

            let newTextFile = File.Create(fighterHtmlDirectory + "\\" + fighterId.ToString("00000") + " " + fighterName.TrimEnd([|' '|]) + ".html")
            use file = new StreamWriter(newTextFile) 
            file.Write(html) 
            file.Close()
        with
            :? System.Net.WebException -> File.AppendAllText("G:\User\WebScraping\Invalid Urls.txt", fighterUrl + "\n")
    }

Upvotes: 3

Related Questions