CODEWITHSUNDEEP
pythonbashbeautifulsoupurlretrieve
user23826167
user23826167

Reputation: 1

BeautifulSoup works when running code from the python command line, but not when running a python script from a shell script

I am interested in extracting data added to a database each day using a cronjob. The cronjob is supposed to run at hourly increments to extract the data by executing a shell script that then runs a python script. The error message I receive when running the shell script to execute the python script results in the following:

for i in '${madis_sources}'
++ python /Users/estrobach/Python_Scripts/Urban_Evaluations/madisData.py  
acarsProfiles
*2024-07-18 2024-07-19 -80 -74 37 43 False
2024-07-18 00:00:00
/Users/estrobach/Data/acarsProfiles/
point
acarsProfiles
False*
Traceback (most recent call last):
  File "/Users/estrobach/Python_Scripts/Urban_Evaluations/madisData.py", line 25, in <module>
    dat=madisImport(Date,database,Type,source,archive)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 21, in madisImport
  File "/opt/anaconda3/lib/python3.11/urllib/request.py", line 241, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
                            ^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/lib/python3.11/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/lib/python3.11/urllib/request.py", line 525, in open
    response = meth(req, response)
               ^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/lib/python3.11/urllib/request.py", line 634, in http_response
    response = self.parent.error(
               ^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/lib/python3.11/urllib/request.py", line 563, in error
    return self._call_chain(*args)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/lib/python3.11/urllib/request.py", line 496, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/opt/anaconda3/lib/python3.11/urllib/request.py", line 643, in **http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found**

The line where the error occurs in madisData.py is related to this command: dat=madisImport(Date,database,Type,source,archive). The exact error in the madisImport.py file is located in line 21 as referenced above which pertains to this line of code: urllib.request.urlretrieve(f, DataDump + fname)

The execution of this line leads to the error response shown above. Interestingly, if I run the python code using the python command line, the error does not occur and I get the data I need. It only occurs when I run the python script using a bash script. Note that the python version is the correct version for both instances.

I expected to get the data I needed from the current day since I'm using the same code to extract archived data (i.e., data before today). I thought that since the present day wasn't over yet, that the code was looking for hours within the present day that had not happened yet. Turns out that wasn't the issue. I also checked for python version consistency. Other than that, I can't really think of what would cause an issue when running the code from a shell script. It works just fine when putting the lines into the python command prompt.

Upvotes: 0

Views: 61

Answers (0)

Related Questions