Pulling log messages from a MySQL table

Clash Royale CLAN TAG#URR8PPP
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty margin-bottom:0;
up vote
3
down vote
favorite
I want to get this code reviewed for a function I wrote that pulls log messages from a MySQL table:
def get_logs_messages(request_id, destination=None):
"""
get logs from deployment table
:param int request_id: request id of selected deployment.
:param str destination: destination site
:return: list of messagesfor selected deployment based on request id.
:rtype: list
"""
conn = MySQLdb.connect(**dbconfig)
query = "SELECT deployment_id "
"from deployment "
"WHERE request_id=request_id "
" dest".format(
request_id=request_id,
dest="AND DESTINATION = 'loc'".format(loc=destination) if destination else "")
cursor = conn.cursor()
cursor.execute(query)
data = cursor.fetchall()
cursor.close()
msgs =
for item in data:
query = "SELECT modified_dt , message from scripts_deployment_logs WHERE deployment_id = ".format(item[0])
cursor = conn.cursor()
cursor.execute(query)
result = cursor.fetchall()
for dt, msg in result:
msgs.append("dt: msg".format(dt=str(dt), msg=msg))
cursor.close()
conn.close()
return msgs
I get a request ID from a web interface which is using a REST API to call this function, and this function retrieves the deployment id based on which it displays the logs. I am not very fluent with the MySQL part so I have used mysql.connector with simple SQL queries.
python performance python-2.7 mysql
add a comment |Â
up vote
3
down vote
favorite
I want to get this code reviewed for a function I wrote that pulls log messages from a MySQL table:
def get_logs_messages(request_id, destination=None):
"""
get logs from deployment table
:param int request_id: request id of selected deployment.
:param str destination: destination site
:return: list of messagesfor selected deployment based on request id.
:rtype: list
"""
conn = MySQLdb.connect(**dbconfig)
query = "SELECT deployment_id "
"from deployment "
"WHERE request_id=request_id "
" dest".format(
request_id=request_id,
dest="AND DESTINATION = 'loc'".format(loc=destination) if destination else "")
cursor = conn.cursor()
cursor.execute(query)
data = cursor.fetchall()
cursor.close()
msgs =
for item in data:
query = "SELECT modified_dt , message from scripts_deployment_logs WHERE deployment_id = ".format(item[0])
cursor = conn.cursor()
cursor.execute(query)
result = cursor.fetchall()
for dt, msg in result:
msgs.append("dt: msg".format(dt=str(dt), msg=msg))
cursor.close()
conn.close()
return msgs
I get a request ID from a web interface which is using a REST API to call this function, and this function retrieves the deployment id based on which it displays the logs. I am not very fluent with the MySQL part so I have used mysql.connector with simple SQL queries.
python performance python-2.7 mysql
add a comment |Â
up vote
3
down vote
favorite
up vote
3
down vote
favorite
I want to get this code reviewed for a function I wrote that pulls log messages from a MySQL table:
def get_logs_messages(request_id, destination=None):
"""
get logs from deployment table
:param int request_id: request id of selected deployment.
:param str destination: destination site
:return: list of messagesfor selected deployment based on request id.
:rtype: list
"""
conn = MySQLdb.connect(**dbconfig)
query = "SELECT deployment_id "
"from deployment "
"WHERE request_id=request_id "
" dest".format(
request_id=request_id,
dest="AND DESTINATION = 'loc'".format(loc=destination) if destination else "")
cursor = conn.cursor()
cursor.execute(query)
data = cursor.fetchall()
cursor.close()
msgs =
for item in data:
query = "SELECT modified_dt , message from scripts_deployment_logs WHERE deployment_id = ".format(item[0])
cursor = conn.cursor()
cursor.execute(query)
result = cursor.fetchall()
for dt, msg in result:
msgs.append("dt: msg".format(dt=str(dt), msg=msg))
cursor.close()
conn.close()
return msgs
I get a request ID from a web interface which is using a REST API to call this function, and this function retrieves the deployment id based on which it displays the logs. I am not very fluent with the MySQL part so I have used mysql.connector with simple SQL queries.
python performance python-2.7 mysql
I want to get this code reviewed for a function I wrote that pulls log messages from a MySQL table:
def get_logs_messages(request_id, destination=None):
"""
get logs from deployment table
:param int request_id: request id of selected deployment.
:param str destination: destination site
:return: list of messagesfor selected deployment based on request id.
:rtype: list
"""
conn = MySQLdb.connect(**dbconfig)
query = "SELECT deployment_id "
"from deployment "
"WHERE request_id=request_id "
" dest".format(
request_id=request_id,
dest="AND DESTINATION = 'loc'".format(loc=destination) if destination else "")
cursor = conn.cursor()
cursor.execute(query)
data = cursor.fetchall()
cursor.close()
msgs =
for item in data:
query = "SELECT modified_dt , message from scripts_deployment_logs WHERE deployment_id = ".format(item[0])
cursor = conn.cursor()
cursor.execute(query)
result = cursor.fetchall()
for dt, msg in result:
msgs.append("dt: msg".format(dt=str(dt), msg=msg))
cursor.close()
conn.close()
return msgs
I get a request ID from a web interface which is using a REST API to call this function, and this function retrieves the deployment id based on which it displays the logs. I am not very fluent with the MySQL part so I have used mysql.connector with simple SQL queries.
python performance python-2.7 mysql
edited Jun 28 at 2:05
Jamalâ¦
30.1k11114225
30.1k11114225
asked Jun 28 at 1:58
Ciasto piekarz
386720
386720
add a comment |Â
add a comment |Â
2 Answers
2
active
oldest
votes
up vote
1
down vote
accepted
Instead of joining the data of two tables in Python, you should une an SQL join query to ask the database do the work for you.
You also should not inject parameters directly into the query and let the driver do it for you by means of parametrized queries.
Lastly, a list-comprehension is better suited to build your output than using append.
Proposed improvements:
def get_logs_messages(request_id, destination=None):
parameters = (request_id,)
query = (
'SELECT logs.modified_dt, logs.message '
'FROM scripts_deployment_logs AS logs '
'LEFT JOIN deployment ON logs.deployment_id = deployment.deployment_id '
'WHERE deployment.request_id = %s'
)
if destination is not None:
query += ' AND deployment.destination = %s'
parameters = (request_id, destination)
conn = MySQLdb.connect(**dbconfig)
cursor = conn.cursor()
cursor.execute(query, parameters)
messages = [
': '.format(dt, message)
for dt, message in cursor.fetchall()
]
cursor.close()
return messages
yes list comprehension is certainly more performance intuitive.
â Ciasto piekarz
Jun 28 at 9:54
add a comment |Â
up vote
1
down vote
In addition to Mathias excelent answer
- You could use context managers, so the connection will close automagically, when you are done. See PEP#343
Don'tstring.formatSQL queries, since this may lead to SQLi vulnerabilities.
from contextlib import closing
def get_logs_messages(request_id, destination=None):
...
with closing(MySQLdb.connect(**dbconfig)) as conn:
with conn as cursor:
cursor.execute(query, parameters)
return [
': '.format(dt, message)
for dt, message in cursor.fetchall()
]
Since this is not part of the DB API 2.0, I wasn't sure ifMySQLdbimplemented such behaviours. Seems like every major DB driver adopted the practice anyways. And now that you have context managers, you can directlyreturnthe list-comprehension.
â Mathias Ettinger
Jun 28 at 19:50
I am usingmysql.connectorso I am doingfrom mysql.connector import MySQLConnection. on using @Ludisposed suggestion I getAttributeError: __exit__after withconn as cursor:
â Ciasto piekarz
Jun 29 at 2:47
1
@Ciastopiekarz You can then changewith conn as cursor:towith closing(conn.cursor()) as cursor:.
â Mathias Ettinger
Jun 29 at 10:14
add a comment |Â
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
1
down vote
accepted
Instead of joining the data of two tables in Python, you should une an SQL join query to ask the database do the work for you.
You also should not inject parameters directly into the query and let the driver do it for you by means of parametrized queries.
Lastly, a list-comprehension is better suited to build your output than using append.
Proposed improvements:
def get_logs_messages(request_id, destination=None):
parameters = (request_id,)
query = (
'SELECT logs.modified_dt, logs.message '
'FROM scripts_deployment_logs AS logs '
'LEFT JOIN deployment ON logs.deployment_id = deployment.deployment_id '
'WHERE deployment.request_id = %s'
)
if destination is not None:
query += ' AND deployment.destination = %s'
parameters = (request_id, destination)
conn = MySQLdb.connect(**dbconfig)
cursor = conn.cursor()
cursor.execute(query, parameters)
messages = [
': '.format(dt, message)
for dt, message in cursor.fetchall()
]
cursor.close()
return messages
yes list comprehension is certainly more performance intuitive.
â Ciasto piekarz
Jun 28 at 9:54
add a comment |Â
up vote
1
down vote
accepted
Instead of joining the data of two tables in Python, you should une an SQL join query to ask the database do the work for you.
You also should not inject parameters directly into the query and let the driver do it for you by means of parametrized queries.
Lastly, a list-comprehension is better suited to build your output than using append.
Proposed improvements:
def get_logs_messages(request_id, destination=None):
parameters = (request_id,)
query = (
'SELECT logs.modified_dt, logs.message '
'FROM scripts_deployment_logs AS logs '
'LEFT JOIN deployment ON logs.deployment_id = deployment.deployment_id '
'WHERE deployment.request_id = %s'
)
if destination is not None:
query += ' AND deployment.destination = %s'
parameters = (request_id, destination)
conn = MySQLdb.connect(**dbconfig)
cursor = conn.cursor()
cursor.execute(query, parameters)
messages = [
': '.format(dt, message)
for dt, message in cursor.fetchall()
]
cursor.close()
return messages
yes list comprehension is certainly more performance intuitive.
â Ciasto piekarz
Jun 28 at 9:54
add a comment |Â
up vote
1
down vote
accepted
up vote
1
down vote
accepted
Instead of joining the data of two tables in Python, you should une an SQL join query to ask the database do the work for you.
You also should not inject parameters directly into the query and let the driver do it for you by means of parametrized queries.
Lastly, a list-comprehension is better suited to build your output than using append.
Proposed improvements:
def get_logs_messages(request_id, destination=None):
parameters = (request_id,)
query = (
'SELECT logs.modified_dt, logs.message '
'FROM scripts_deployment_logs AS logs '
'LEFT JOIN deployment ON logs.deployment_id = deployment.deployment_id '
'WHERE deployment.request_id = %s'
)
if destination is not None:
query += ' AND deployment.destination = %s'
parameters = (request_id, destination)
conn = MySQLdb.connect(**dbconfig)
cursor = conn.cursor()
cursor.execute(query, parameters)
messages = [
': '.format(dt, message)
for dt, message in cursor.fetchall()
]
cursor.close()
return messages
Instead of joining the data of two tables in Python, you should une an SQL join query to ask the database do the work for you.
You also should not inject parameters directly into the query and let the driver do it for you by means of parametrized queries.
Lastly, a list-comprehension is better suited to build your output than using append.
Proposed improvements:
def get_logs_messages(request_id, destination=None):
parameters = (request_id,)
query = (
'SELECT logs.modified_dt, logs.message '
'FROM scripts_deployment_logs AS logs '
'LEFT JOIN deployment ON logs.deployment_id = deployment.deployment_id '
'WHERE deployment.request_id = %s'
)
if destination is not None:
query += ' AND deployment.destination = %s'
parameters = (request_id, destination)
conn = MySQLdb.connect(**dbconfig)
cursor = conn.cursor()
cursor.execute(query, parameters)
messages = [
': '.format(dt, message)
for dt, message in cursor.fetchall()
]
cursor.close()
return messages
edited Jun 28 at 19:52
answered Jun 28 at 9:49
Mathias Ettinger
21.7k32875
21.7k32875
yes list comprehension is certainly more performance intuitive.
â Ciasto piekarz
Jun 28 at 9:54
add a comment |Â
yes list comprehension is certainly more performance intuitive.
â Ciasto piekarz
Jun 28 at 9:54
yes list comprehension is certainly more performance intuitive.
â Ciasto piekarz
Jun 28 at 9:54
yes list comprehension is certainly more performance intuitive.
â Ciasto piekarz
Jun 28 at 9:54
add a comment |Â
up vote
1
down vote
In addition to Mathias excelent answer
- You could use context managers, so the connection will close automagically, when you are done. See PEP#343
Don'tstring.formatSQL queries, since this may lead to SQLi vulnerabilities.
from contextlib import closing
def get_logs_messages(request_id, destination=None):
...
with closing(MySQLdb.connect(**dbconfig)) as conn:
with conn as cursor:
cursor.execute(query, parameters)
return [
': '.format(dt, message)
for dt, message in cursor.fetchall()
]
Since this is not part of the DB API 2.0, I wasn't sure ifMySQLdbimplemented such behaviours. Seems like every major DB driver adopted the practice anyways. And now that you have context managers, you can directlyreturnthe list-comprehension.
â Mathias Ettinger
Jun 28 at 19:50
I am usingmysql.connectorso I am doingfrom mysql.connector import MySQLConnection. on using @Ludisposed suggestion I getAttributeError: __exit__after withconn as cursor:
â Ciasto piekarz
Jun 29 at 2:47
1
@Ciastopiekarz You can then changewith conn as cursor:towith closing(conn.cursor()) as cursor:.
â Mathias Ettinger
Jun 29 at 10:14
add a comment |Â
up vote
1
down vote
In addition to Mathias excelent answer
- You could use context managers, so the connection will close automagically, when you are done. See PEP#343
Don'tstring.formatSQL queries, since this may lead to SQLi vulnerabilities.
from contextlib import closing
def get_logs_messages(request_id, destination=None):
...
with closing(MySQLdb.connect(**dbconfig)) as conn:
with conn as cursor:
cursor.execute(query, parameters)
return [
': '.format(dt, message)
for dt, message in cursor.fetchall()
]
Since this is not part of the DB API 2.0, I wasn't sure ifMySQLdbimplemented such behaviours. Seems like every major DB driver adopted the practice anyways. And now that you have context managers, you can directlyreturnthe list-comprehension.
â Mathias Ettinger
Jun 28 at 19:50
I am usingmysql.connectorso I am doingfrom mysql.connector import MySQLConnection. on using @Ludisposed suggestion I getAttributeError: __exit__after withconn as cursor:
â Ciasto piekarz
Jun 29 at 2:47
1
@Ciastopiekarz You can then changewith conn as cursor:towith closing(conn.cursor()) as cursor:.
â Mathias Ettinger
Jun 29 at 10:14
add a comment |Â
up vote
1
down vote
up vote
1
down vote
In addition to Mathias excelent answer
- You could use context managers, so the connection will close automagically, when you are done. See PEP#343
Don'tstring.formatSQL queries, since this may lead to SQLi vulnerabilities.
from contextlib import closing
def get_logs_messages(request_id, destination=None):
...
with closing(MySQLdb.connect(**dbconfig)) as conn:
with conn as cursor:
cursor.execute(query, parameters)
return [
': '.format(dt, message)
for dt, message in cursor.fetchall()
]
In addition to Mathias excelent answer
- You could use context managers, so the connection will close automagically, when you are done. See PEP#343
Don'tstring.formatSQL queries, since this may lead to SQLi vulnerabilities.
from contextlib import closing
def get_logs_messages(request_id, destination=None):
...
with closing(MySQLdb.connect(**dbconfig)) as conn:
with conn as cursor:
cursor.execute(query, parameters)
return [
': '.format(dt, message)
for dt, message in cursor.fetchall()
]
edited Jun 28 at 21:28
answered Jun 28 at 11:04
Ludisposed
5,68621656
5,68621656
Since this is not part of the DB API 2.0, I wasn't sure ifMySQLdbimplemented such behaviours. Seems like every major DB driver adopted the practice anyways. And now that you have context managers, you can directlyreturnthe list-comprehension.
â Mathias Ettinger
Jun 28 at 19:50
I am usingmysql.connectorso I am doingfrom mysql.connector import MySQLConnection. on using @Ludisposed suggestion I getAttributeError: __exit__after withconn as cursor:
â Ciasto piekarz
Jun 29 at 2:47
1
@Ciastopiekarz You can then changewith conn as cursor:towith closing(conn.cursor()) as cursor:.
â Mathias Ettinger
Jun 29 at 10:14
add a comment |Â
Since this is not part of the DB API 2.0, I wasn't sure ifMySQLdbimplemented such behaviours. Seems like every major DB driver adopted the practice anyways. And now that you have context managers, you can directlyreturnthe list-comprehension.
â Mathias Ettinger
Jun 28 at 19:50
I am usingmysql.connectorso I am doingfrom mysql.connector import MySQLConnection. on using @Ludisposed suggestion I getAttributeError: __exit__after withconn as cursor:
â Ciasto piekarz
Jun 29 at 2:47
1
@Ciastopiekarz You can then changewith conn as cursor:towith closing(conn.cursor()) as cursor:.
â Mathias Ettinger
Jun 29 at 10:14
Since this is not part of the DB API 2.0, I wasn't sure if
MySQLdb implemented such behaviours. Seems like every major DB driver adopted the practice anyways. And now that you have context managers, you can directly return the list-comprehension.â Mathias Ettinger
Jun 28 at 19:50
Since this is not part of the DB API 2.0, I wasn't sure if
MySQLdb implemented such behaviours. Seems like every major DB driver adopted the practice anyways. And now that you have context managers, you can directly return the list-comprehension.â Mathias Ettinger
Jun 28 at 19:50
I am using
mysql.connector so I am doing from mysql.connector import MySQLConnection. on using @Ludisposed suggestion I get AttributeError: __exit__ after with conn as cursor:â Ciasto piekarz
Jun 29 at 2:47
I am using
mysql.connector so I am doing from mysql.connector import MySQLConnection. on using @Ludisposed suggestion I get AttributeError: __exit__ after with conn as cursor:â Ciasto piekarz
Jun 29 at 2:47
1
1
@Ciastopiekarz You can then change
with conn as cursor: to with closing(conn.cursor()) as cursor:.â Mathias Ettinger
Jun 29 at 10:14
@Ciastopiekarz You can then change
with conn as cursor: to with closing(conn.cursor()) as cursor:.â Mathias Ettinger
Jun 29 at 10:14
add a comment |Â
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fcodereview.stackexchange.com%2fquestions%2f197392%2fpulling-log-messages-from-a-mysql-table%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password