ValueError: invalid literal for int( ) with base 10: '262,400,000'

Question

I'm working on a Django application which fetches JSON data from an API and stores it in PostgreSQL database. But while migrating the app I'm getting this error:

ValueError: invalid literal for int() with base 10: '262,400,000'

Here's the traceback:

Traceback (most recent call last):
  File "manage.py", line 22, in <module>
    execute_from_command_line(sys.argv)
  File "/python/lib/python3.6/site-packages/django/core/management/__init__.py", line 363, in execute_from_command_line
    utility.execute()
  File "/python/lib/python3.6/site-packages/django/core/management/__init__.py", line 355, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/python/lib/python3.6/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/python/lib/python3.6/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/app/aggregator/WorldBank/management/commands/fetch_wb.py", line 60, in handle
    Projects.objects.create(**pdata)
  File "/python/lib/python3.6/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/python/lib/python3.6/site-packages/django/db/models/query.py", line 394, in create
    obj.save(force_insert=True, using=self.db)
  File "/python/lib/python3.6/site-packages/django/db/models/base.py", line 807, in save
    force_update=force_update, update_fields=update_fields)
  File "/python/lib/python3.6/site-packages/django/db/models/base.py", line 837, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/python/lib/python3.6/site-packages/django/db/models/base.py", line 923, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/python/lib/python3.6/site-packages/django/db/models/base.py", line 962, in _do_insert
    using=using, raw=raw)
  File "/python/lib/python3.6/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/python/lib/python3.6/site-packages/django/db/models/query.py", line 1076, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/python/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1106, in execute_sql
    for sql, params in self.as_sql():
  File "/python/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1059, in as_sql
    for obj in self.query.objs
  File "/python/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1059, in <listcomp>
    for obj in self.query.objs
  File "/python/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1058, in <listcomp>
    [self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields]
  File "/python/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 998, in prepare_value
    value = field.get_db_prep_save(value, connection=self.connection)
  File "/python/lib/python3.6/site-packages/django/db/models/fields/__init__.py", line 770, in get_db_prep_save
    prepared=False)
  File "/python/lib/python3.6/site-packages/django/db/models/fields/__init__.py", line 762, in get_db_prep_value
    value = self.get_prep_value(value)
  File "/python/lib/python3.6/site-packages/django/db/models/fields/__init__.py", line 1853, in get_prep_value
    return int(value)
ValueError: invalid literal for int() with base 10: '262,400,000'

How to fix this problem?

Here's my code for models.py:

from django.db import models
from django.contrib.postgres.fields import JSONField
from django.core.validators import URLValidator

class Projects(models.Model):
        project_id=models.CharField(max_length=50)
        projectfinancialtype=models.CharField(max_length=10)
        regionname=models.CharField(max_length=50)
        prodline=models.CharField(max_length=50)
        lendinginstr=models.CharField(max_length=100)
        lendinginstrtype=models.CharField(max_length=10)
        envassesmentcategorycode=models.CharField(max_length=10)
        supplementprojectflg=models.CharField(max_length=10)
        productlinetype=models.CharField(max_length=10)
        projectstatusdisplay=models.CharField(max_length=10)
        status=models.CharField(max_length=10)
        project_name=models.CharField(max_length=100)
        board_approval_month=models.CharField(max_length=10)
        approvalfy=models.CharField(max_length=9)
        boardapprovaldate=models.DateTimeField(auto_now=False, auto_now_add=False)
        closingdate=models.DateTimeField(auto_now=False, auto_now_add=False)
        lendprojectcost=models.IntegerField()
        ibrdcommamt=models.IntegerField()
        idacommamt=models.IntegerField()
        totalamt=models.IntegerField()
        grantamt=models.IntegerField()
        borrower=models.CharField(max_length=20)
        impagency=models.CharField(max_length=20)
        countryshortname=models.IntegerField()
        prodlinetext=models.CharField(max_length=20)
        project_abstract=JSONField()
        p2a_flag=models.CharField(max_length=10)
        p2a_updated_date=models.DateTimeField(auto_now=False, auto_now_add=False)
        country_namecode=models.CharField(max_length=100)
        projectinfo=models.TextField()
        sector1=JSONField()
        theme1=JSONField()
        theme2=JSONField()
        theme3=JSONField()
        url=models.TextField(validators=[URLValidator()])
        totalcommamt=models.IntegerField()
        mjthemecode=models.CharField(max_length=20)
        ccsa_practice_name=models.CharField(max_length=100)
        combined_practice_code=models.CharField(max_length=100)
        theme_namecode=JSONField()#ArrayField(JSONField(), blank=True)
        themecode=models.IntegerField()
        countrycode=models.CharField(max_length=10)
        countryname=models.CharField(max_length=100)
        docty=models.TextField()
        mjtheme_namecode=JSONField() #ArrayField(JSONField(), blank=True)
        source=models.CharField(max_length=10)
        projectdocs=JSONField() #ArrayField(JSONField(), blank=True)
        combined_practice_name=models.CharField(max_length=100)
        mjtheme=JSONField() #ArrayField(JSONField(), blank=True)
        ccsa_practice_code=models.CharField(max_length=100)
        teamleaderupi=models.IntegerField()
        locations=JSONField()
        facets=JSONField()

And here's the code for fetch.py file which is stored under /management/commands/fetch.py:

import requests
from django.core.management.base import BaseCommand
from aggregator.WorldBank.models import Projects

class Command(BaseCommand):
    def handle(self, **options):
        response = requests.get("https://search.worldbank.org/api/v2/projects?format=json&countryshortname_exact=India&source=IBRD&kw=N&rows=776")
        data = response.json()
        projects = data.get('projects')

        for pdata in projects.values():
            pdata['project_id'] = pdata.pop('id', None)
            pdata['countryshortname'] = pdata.pop('countryshortname', None)
            pdata['projectfinancialtype'] = pdata.pop('projectfinancialtype',None)
            pdata['regionname'] = pdata.pop('regionname',None)
            pdata['prodline'] = pdata.pop('prodline',None)
            pdata['lendinginstr'] = pdata.pop('lendinginstr',None)
            pdata['lendinginstrtype'] = pdata.pop('lendinginstrtype',None)
            pdata['envassesmentcategorycode']=pdata.pop('envassesmentcategorycode',None)
            pdata['supplementprojectflg']=pdata.pop('supplementprojectflg',None)
            pdata['productlinetype']=pdata.pop('productlinetype',None)
            pdata['projectstatusdisplay']=pdata.pop('projectstatusdisplay',None)
            pdata['status']=pdata.pop('status',None)
            pdata['project_name']=pdata.pop('project_name',None)
            pdata['board_approval_month']=pdata.pop('board_approval_month',None)
            pdata['approvalfy']=pdata.pop('approvalfy',None)
            pdata['boardapprovaldate']=pdata.pop('boardapprovaldate',None)
            pdata['closingdate']=pdata.pop('closingdate',None)
            pdata['lendprojectcost']=pdata.pop('lendprojectcost',None)
            pdata['ibrdcommamt']=pdata.pop('ibrdcommamt',None)
            pdata['idacommamt']=pdata.pop('idacommamt',None)
            pdata['totalamt']=pdata.pop('totalamt',None)
            pdata['grantamt']=pdata.pop('grantamt',None)
            pdata['teamleaderupi']=pdata.pop('teamleaderupi',None)
            pdata['prodlinetext']=pdata.pop('prodlinetext',None)
            pdata['project_abstract']=pdata.pop('project_abstract',None)
            pdata['p2a_flag']=pdata.pop('p2a_flag',None)
            pdata['p2a_updated_date']=pdata.pop('p2a_updated_date',None)
            pdata['country_namecode']=pdata.pop('country_namecode',None)
            pdata['projectinfo']=pdata.pop('projectinfo',None)
            pdata['sector1']=pdata.pop('sector1',None)
            pdata['theme1']=pdata.pop('theme1',None)
            pdata['theme2']=pdata.pop('theme2',None)
            pdata['theme3']=pdata.pop('theme3',None)
            pdata['url']=pdata.pop('url',None)
            pdata['totalcommamt']=pdata.pop('totalcommamt',None)
            pdata['mjthemecode']=pdata.pop('mjthemecode',None)
            pdata['theme_namecode']=pdata.pop('theme_namecode',None)
            pdata['themecode']=pdata.pop('themecode',None)
            pdata['countrycode']=pdata.pop('countrycode',None)
            pdata['countryname']=pdata.pop('countryname',None)
            pdata['docty']=pdata.pop('docty',None)
            pdata['mjtheme_namecode']=pdata.pop('mjtheme_namecode',None)
            pdata['source']=pdata.pop('source',None)
            pdata['projectdocs']=pdata.pop('projectdocs',None)
            pdata['mjtheme']=pdata.pop('mjtheme',None)
            pdata['locations']=pdata.pop('locations',None)
            pdata['facets']=pdata.pop('facets',None)
            # other columns need to be ranamed

            Projects.objects.create(**pdata)

This is the API URL from which I want to store the JSON responce into the postgres database:API URL

Answer 1

If you change the db table datatype for all "amounts" to money then you should be able to insert those fields as a string via python and keep the commas.

.

The saying goes, "If you have a big, beautiful hammer..."

So, I imported the API data into a database table using sql functions and then this is just one of the ways that the data can be accessed:

http://dbfiddle.uk/?rdbms=postgres_10&fiddle=784a69df6106d5997875c798354a5777

some useful links for this nail above :)

https://www.postgresql.org/docs/current/static/sql-createtype.html https://www.postgresql.org/docs/current/static/functions-json.html

Answer 2

Replace the commas with _ (python 3 supports underscores):

num.replace(',', '_')

Change them back later.