CODEWITHSUNDEEP
pythonsqlsql-server
Nishant
Nishant

Reputation: 21914

Why does my query break when it is parameterized?

I have 2 tables - Sales and Product. Sales can store the product as Idn or Name (legacy design) and the Type column specifies the actual type associated to it. Product etc. is a subset table that is joined into this table to get the real data. (In this example, Product is a table that stores Idn's to demonstrate the issue.)

Sales

|------------|--------------------|----------------|
|    Idn     |  Product Idn/Name  |     Type       |
|------------|--------------------|----------------|
|     1      |          1         |     Number     |
|------------|--------------------|----- ----------|
|     2      |       Colgate      |      Word      |
|------------|--------------------|----------------|

Product (Idn)

|------------|------------------|
|    Idn     |    Some Info     |
|------------|------------------|
|     1      |       ...        |
|------------|------------------|

Normally, you should not join these tables on Product Idn because it has mixed data; but if you select the rows where LHS matches RHS, it works fine (1). For example, if Product is a table that stores Idns, the following query fails:

SELECT * from sales JOIN product on sales.pid = product.idn

but the following query works:

SELECT * from sales JOIN product on sales.pid = product.idn WHERE type = 'Number'

This also works as expected in Python 2 + SQLAlchemy + PyODBC as well. However, when I try this in Python 3 + SQLAlchemy + PyODBC, it gives me a datatype conversion error and it only happens when the query is parameterized!

Now if I make it u'number' in Python 2, it breaks there as well; and b'number' works in Python 3! I am guessing there is some issue with Unicode conversion. Is it trying to guess encoding and doing something wrong? Can I fix this by being more explicit?

The error received is:

Traceback (most recent call last):
  File "reproduce.py", line 59, in <module>
    print(cursor.execute(select_parametrized, ('number', 1)).fetchall())
pyodbc.ProgrammingError: ('42000', '[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Error converting data type varchar to numeric. (8114) (SQLFetch)

What could be the issue here and is there any good ways to side-step the problem without doing things like convert (because it worked in a previous version)?

Here is a query that can be used to reproduce this issue with no side-effects (needs SQLAlchemy and PyODBC):

import sqlalchemy
import sqlalchemy.orm

create_tables = """
CREATE TABLE products(
    idn NUMERIC(9) PRIMARY KEY
);
CREATE TABLE sales(
    idn NUMERIC(9) PRIMARY KEY,
    pid VARCHAR(50) NOT NULL,
    type VARCHAR(10) NOT NULL
);
"""

check_tables_exist = """   
SELECT * FROM products;
SELECT * FROM sales;
"""

insert_values = """
INSERT INTO products (idn) values (1);
INSERT INTO sales (idn, pid, type) values (1, 1, 'number');
INSERT INTO sales (idn, pid, type) values (2, 'Colgate', 'word');
"""

select_adhoc = """
SELECT * FROM products
JOIN sales ON products.idn = sales.pid
AND sales.type = 'number'
WHERE products.idn in (1);
"""

select_parametrized = """
SELECT * FROM products
JOIN sales ON products.idn = sales.pid
AND sales.type = ?
WHERE products.idn in (?);
"""

delete_tables = """
DROP TABLE products;
DROP TABLE sales;
"""

engine = sqlalchemy.create_engine('mssql+pyodbc://user:password@dsn')
connection = engine.connect()
cursor = engine.raw_connection().cursor()

Session = sqlalchemy.orm.sessionmaker(bind=connection)
session = Session()

session.execute(create_tables)

try:
    session.execute(check_tables_exist)
    session.execute(insert_values)
    session.commit()
    print(cursor.execute(select_adhoc).fetchall())
    print(cursor.execute(select_parametrized, ('number', 1)).fetchall())
finally:
    session.execute(delete_tables)
    session.commit()

1. This was a wrong assumption. It worked by chance - SQL's execution plan gave priority to this condition as explained here. It didn't do that when it became NVARCHAR.

Upvotes: 3

Views: 466

Answers (1)

Serkan Arslan
Serkan Arslan

Reputation: 13393

SQLAlchemy generates this SQL script with your non-parameterized query (select_adhoc):

SELECT * FROM products
JOIN sales ON products.idn = sales.pid
AND sales.type = 'number'
WHERE products.idn in (1);

But with the parameterized query (select_parametrized), it generates this: (I checked from SQL Server Profiler.)

declare @p1 int
set @p1=NULL
exec sp_prepexec @p1 output,N'@P1 nvarchar(12),@P2 int',N'
SELECT * FROM products
INNER JOIN sales ON products.idn = sales.pid 
    AND sales.type = @P1
WHERE products.idn in (@P2);
',N'number',1
select @p1

If you try this on SQL Server you will get the same exception:

Msg 8114, Level 16, State 5, Line 32 Error converting data type varchar to numeric.

The problem is at the @P1 parameter declaration -- it makes an implicit conversion to varchar (the type of sales.type) and that causes this problem. Probably Python 2 generates varchar?

If you change your query like this it will work correctly; or you need to change the type of sales.type to nvarchar.

select_parametrized = """
SELECT * FROM products
INNER JOIN sales ON products.idn = sales.pid 
    AND sales.type = CAST(? AS VARCHAR(50))
WHERE products.idn in (?);
"""

Upvotes: 2

Related Questions