Accessing a subset of keys from a nested dictionary in Python
Clash Royale CLAN TAG#URR8PPP
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty margin-bottom:0;
up vote
2
down vote
favorite
I am trying to access data from Coinmarketcap API in Python and came up with the following code:
def fetch_coin_prices(**kwargs):
"""Retrieve cryptocurrency data from CoinMarketCap and return a dictionary
containing coin names with their current prices.
Keyword arguments to this function are mapped to the CoinMarketCap API,
refer to their documentation for their meaning:
https://coinmarketcap.com/api/
"""
response = requests.get(
'https://api.coinmarketcap.com/v2/ticker/',
params=kwargs
)
response.raise_for_status()
coin_data = response.json()
return coin_data.get('data', ).values()
data = fetch_coin_prices(limit=100, start=0, sort='id')
I am trying to write this data to a Redis HMSET that accepts a dictionary and I only want the properties 'id', 'name', 'symbol', 'rank', 'price' and 'volume_24h'. The price and volume properties are nested and therefore I haven't found a way to get everything in 1 go. My redis hmset needs to store data in key value form where key is the coin id and value is the CSV of the current coin.
hmset =
fieldnames = ['id', 'name', 'symbol', 'rank' ,'price', 'volume_24h']
for row in data:
subset =
subset['id'] = row.get('id', None)
subset['name'] = row.get('name', None)
subset['symbol'] = row.get('symbol', None)
subset['rank'] = row.get('rank', None)
subset['price'] = row.get('quotes', ).get('USD', ).get('price', None)
subset['volume_24h'] = row.get('quotes', ).get('USD', ).get('volume_24h', None)
if all([subset[key] for key in subset]):
csv_string = io.StringIO()
csv_writer = csv.DictWriter(csv_string, fieldnames=fieldnames, extrasaction='ignore' , lineterminator=':')
csv_writer.writerows(data_subset)
hmset[subset['id']] = csv_string.getvalue()
This is what I have come up with so far but it looks very ugly in my opinion to access keys. Is there a better way to do this?
python python-3.x dictionary redis
add a comment |Â
up vote
2
down vote
favorite
I am trying to access data from Coinmarketcap API in Python and came up with the following code:
def fetch_coin_prices(**kwargs):
"""Retrieve cryptocurrency data from CoinMarketCap and return a dictionary
containing coin names with their current prices.
Keyword arguments to this function are mapped to the CoinMarketCap API,
refer to their documentation for their meaning:
https://coinmarketcap.com/api/
"""
response = requests.get(
'https://api.coinmarketcap.com/v2/ticker/',
params=kwargs
)
response.raise_for_status()
coin_data = response.json()
return coin_data.get('data', ).values()
data = fetch_coin_prices(limit=100, start=0, sort='id')
I am trying to write this data to a Redis HMSET that accepts a dictionary and I only want the properties 'id', 'name', 'symbol', 'rank', 'price' and 'volume_24h'. The price and volume properties are nested and therefore I haven't found a way to get everything in 1 go. My redis hmset needs to store data in key value form where key is the coin id and value is the CSV of the current coin.
hmset =
fieldnames = ['id', 'name', 'symbol', 'rank' ,'price', 'volume_24h']
for row in data:
subset =
subset['id'] = row.get('id', None)
subset['name'] = row.get('name', None)
subset['symbol'] = row.get('symbol', None)
subset['rank'] = row.get('rank', None)
subset['price'] = row.get('quotes', ).get('USD', ).get('price', None)
subset['volume_24h'] = row.get('quotes', ).get('USD', ).get('volume_24h', None)
if all([subset[key] for key in subset]):
csv_string = io.StringIO()
csv_writer = csv.DictWriter(csv_string, fieldnames=fieldnames, extrasaction='ignore' , lineterminator=':')
csv_writer.writerows(data_subset)
hmset[subset['id']] = csv_string.getvalue()
This is what I have come up with so far but it looks very ugly in my opinion to access keys. Is there a better way to do this?
python python-3.x dictionary redis
2
what do you mean exactly withThe price and volume properties are nested
. please share some example data, and what you expect
â Maarten Fabré
Jun 26 at 7:40
@MaartenFabré thank you for reviewing this question, at the very top I have added a link to the coinmarketcap API containing real data
â PirateApp
Jun 26 at 9:08
add a comment |Â
up vote
2
down vote
favorite
up vote
2
down vote
favorite
I am trying to access data from Coinmarketcap API in Python and came up with the following code:
def fetch_coin_prices(**kwargs):
"""Retrieve cryptocurrency data from CoinMarketCap and return a dictionary
containing coin names with their current prices.
Keyword arguments to this function are mapped to the CoinMarketCap API,
refer to their documentation for their meaning:
https://coinmarketcap.com/api/
"""
response = requests.get(
'https://api.coinmarketcap.com/v2/ticker/',
params=kwargs
)
response.raise_for_status()
coin_data = response.json()
return coin_data.get('data', ).values()
data = fetch_coin_prices(limit=100, start=0, sort='id')
I am trying to write this data to a Redis HMSET that accepts a dictionary and I only want the properties 'id', 'name', 'symbol', 'rank', 'price' and 'volume_24h'. The price and volume properties are nested and therefore I haven't found a way to get everything in 1 go. My redis hmset needs to store data in key value form where key is the coin id and value is the CSV of the current coin.
hmset =
fieldnames = ['id', 'name', 'symbol', 'rank' ,'price', 'volume_24h']
for row in data:
subset =
subset['id'] = row.get('id', None)
subset['name'] = row.get('name', None)
subset['symbol'] = row.get('symbol', None)
subset['rank'] = row.get('rank', None)
subset['price'] = row.get('quotes', ).get('USD', ).get('price', None)
subset['volume_24h'] = row.get('quotes', ).get('USD', ).get('volume_24h', None)
if all([subset[key] for key in subset]):
csv_string = io.StringIO()
csv_writer = csv.DictWriter(csv_string, fieldnames=fieldnames, extrasaction='ignore' , lineterminator=':')
csv_writer.writerows(data_subset)
hmset[subset['id']] = csv_string.getvalue()
This is what I have come up with so far but it looks very ugly in my opinion to access keys. Is there a better way to do this?
python python-3.x dictionary redis
I am trying to access data from Coinmarketcap API in Python and came up with the following code:
def fetch_coin_prices(**kwargs):
"""Retrieve cryptocurrency data from CoinMarketCap and return a dictionary
containing coin names with their current prices.
Keyword arguments to this function are mapped to the CoinMarketCap API,
refer to their documentation for their meaning:
https://coinmarketcap.com/api/
"""
response = requests.get(
'https://api.coinmarketcap.com/v2/ticker/',
params=kwargs
)
response.raise_for_status()
coin_data = response.json()
return coin_data.get('data', ).values()
data = fetch_coin_prices(limit=100, start=0, sort='id')
I am trying to write this data to a Redis HMSET that accepts a dictionary and I only want the properties 'id', 'name', 'symbol', 'rank', 'price' and 'volume_24h'. The price and volume properties are nested and therefore I haven't found a way to get everything in 1 go. My redis hmset needs to store data in key value form where key is the coin id and value is the CSV of the current coin.
hmset =
fieldnames = ['id', 'name', 'symbol', 'rank' ,'price', 'volume_24h']
for row in data:
subset =
subset['id'] = row.get('id', None)
subset['name'] = row.get('name', None)
subset['symbol'] = row.get('symbol', None)
subset['rank'] = row.get('rank', None)
subset['price'] = row.get('quotes', ).get('USD', ).get('price', None)
subset['volume_24h'] = row.get('quotes', ).get('USD', ).get('volume_24h', None)
if all([subset[key] for key in subset]):
csv_string = io.StringIO()
csv_writer = csv.DictWriter(csv_string, fieldnames=fieldnames, extrasaction='ignore' , lineterminator=':')
csv_writer.writerows(data_subset)
hmset[subset['id']] = csv_string.getvalue()
This is what I have come up with so far but it looks very ugly in my opinion to access keys. Is there a better way to do this?
python python-3.x dictionary redis
edited Jun 26 at 2:53
Jamalâ¦
30.1k11114225
30.1k11114225
asked Jun 26 at 2:39
PirateApp
1316
1316
2
what do you mean exactly withThe price and volume properties are nested
. please share some example data, and what you expect
â Maarten Fabré
Jun 26 at 7:40
@MaartenFabré thank you for reviewing this question, at the very top I have added a link to the coinmarketcap API containing real data
â PirateApp
Jun 26 at 9:08
add a comment |Â
2
what do you mean exactly withThe price and volume properties are nested
. please share some example data, and what you expect
â Maarten Fabré
Jun 26 at 7:40
@MaartenFabré thank you for reviewing this question, at the very top I have added a link to the coinmarketcap API containing real data
â PirateApp
Jun 26 at 9:08
2
2
what do you mean exactly with
The price and volume properties are nested
. please share some example data, and what you expectâ Maarten Fabré
Jun 26 at 7:40
what do you mean exactly with
The price and volume properties are nested
. please share some example data, and what you expectâ Maarten Fabré
Jun 26 at 7:40
@MaartenFabré thank you for reviewing this question, at the very top I have added a link to the coinmarketcap API containing real data
â PirateApp
Jun 26 at 9:08
@MaartenFabré thank you for reviewing this question, at the very top I have added a link to the coinmarketcap API containing real data
â PirateApp
Jun 26 at 9:08
add a comment |Â
2 Answers
2
active
oldest
votes
up vote
2
down vote
accepted
The way using dict.get
certainly works, but can become a bit unreadable if you need to chain them.
An alternative is to use exception handling in this case.
subset['volume_24h'] = row.get('quotes', ).get('USD', ).get('volume_24h', None)
is equivalent to
try:
subset['volume_24h'] = row['quotes']['USD']['volume_24h']
except KeyError:
subset['volume_24h'] = None
(This would be a bit neater if PEP-463 had not been rejected:)
subset['volume_24h'] = row['quotes']['USD']['volume_24h'] except KeyError: None
upvoted! thank you for the analysis, one curious question is how do you manage keynames like quotes USD price etc in your code, do you hardcode them, do you have them as environment variables so that you can modify them, some other technique perhaps?
â PirateApp
Jun 26 at 11:14
1
@PirateApp: That depends. If it is one-off code just hard-code them. On the other hand if the possibility exists that you might have some other data source with different structure, you might need to outsource it to a class that gets the data and translates it into some canonical format.
â Graipher
Jun 26 at 11:15
2
Damn, PEP-463 would have made so much Python code sick. There'd also be no need forget
with it too... :O
â Peilonrayz
Jun 26 at 11:25
add a comment |Â
up vote
4
down vote
I previously have written a helper for this, as I've come across it too. If you pass the object and the keys to the object then you can do the same.
def get_nested(obj, keys)
try:
for key in keys:
obj = obj[key]
except KeyError:
return None
return obj
This makes usage:
subset['price'] = get_nested(row, 'quotes USD price'.split())
This allows you to extend the keys argument to a minilanguage for other common methods you'd want to apply to returned objects. One case I had was since JavaScript doesn't like maps much (not objects) some APIs tend to convert dictionaries to arrays, and so I had to search a list of objects for which object I wanted, as I couldn't index the list as I'd get incorrect data at times.
upvoted! thank you for sharing code! i searched for a few questions right here on codereview on flattening nested json and i saw there are a lot of intricacies involved like handling empty keys and any hashable type as opposed to mere string keys, this approach is something i never imagined :)
â PirateApp
Jun 26 at 11:40
add a comment |Â
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
2
down vote
accepted
The way using dict.get
certainly works, but can become a bit unreadable if you need to chain them.
An alternative is to use exception handling in this case.
subset['volume_24h'] = row.get('quotes', ).get('USD', ).get('volume_24h', None)
is equivalent to
try:
subset['volume_24h'] = row['quotes']['USD']['volume_24h']
except KeyError:
subset['volume_24h'] = None
(This would be a bit neater if PEP-463 had not been rejected:)
subset['volume_24h'] = row['quotes']['USD']['volume_24h'] except KeyError: None
upvoted! thank you for the analysis, one curious question is how do you manage keynames like quotes USD price etc in your code, do you hardcode them, do you have them as environment variables so that you can modify them, some other technique perhaps?
â PirateApp
Jun 26 at 11:14
1
@PirateApp: That depends. If it is one-off code just hard-code them. On the other hand if the possibility exists that you might have some other data source with different structure, you might need to outsource it to a class that gets the data and translates it into some canonical format.
â Graipher
Jun 26 at 11:15
2
Damn, PEP-463 would have made so much Python code sick. There'd also be no need forget
with it too... :O
â Peilonrayz
Jun 26 at 11:25
add a comment |Â
up vote
2
down vote
accepted
The way using dict.get
certainly works, but can become a bit unreadable if you need to chain them.
An alternative is to use exception handling in this case.
subset['volume_24h'] = row.get('quotes', ).get('USD', ).get('volume_24h', None)
is equivalent to
try:
subset['volume_24h'] = row['quotes']['USD']['volume_24h']
except KeyError:
subset['volume_24h'] = None
(This would be a bit neater if PEP-463 had not been rejected:)
subset['volume_24h'] = row['quotes']['USD']['volume_24h'] except KeyError: None
upvoted! thank you for the analysis, one curious question is how do you manage keynames like quotes USD price etc in your code, do you hardcode them, do you have them as environment variables so that you can modify them, some other technique perhaps?
â PirateApp
Jun 26 at 11:14
1
@PirateApp: That depends. If it is one-off code just hard-code them. On the other hand if the possibility exists that you might have some other data source with different structure, you might need to outsource it to a class that gets the data and translates it into some canonical format.
â Graipher
Jun 26 at 11:15
2
Damn, PEP-463 would have made so much Python code sick. There'd also be no need forget
with it too... :O
â Peilonrayz
Jun 26 at 11:25
add a comment |Â
up vote
2
down vote
accepted
up vote
2
down vote
accepted
The way using dict.get
certainly works, but can become a bit unreadable if you need to chain them.
An alternative is to use exception handling in this case.
subset['volume_24h'] = row.get('quotes', ).get('USD', ).get('volume_24h', None)
is equivalent to
try:
subset['volume_24h'] = row['quotes']['USD']['volume_24h']
except KeyError:
subset['volume_24h'] = None
(This would be a bit neater if PEP-463 had not been rejected:)
subset['volume_24h'] = row['quotes']['USD']['volume_24h'] except KeyError: None
The way using dict.get
certainly works, but can become a bit unreadable if you need to chain them.
An alternative is to use exception handling in this case.
subset['volume_24h'] = row.get('quotes', ).get('USD', ).get('volume_24h', None)
is equivalent to
try:
subset['volume_24h'] = row['quotes']['USD']['volume_24h']
except KeyError:
subset['volume_24h'] = None
(This would be a bit neater if PEP-463 had not been rejected:)
subset['volume_24h'] = row['quotes']['USD']['volume_24h'] except KeyError: None
answered Jun 26 at 10:50
Graipher
20.4k42981
20.4k42981
upvoted! thank you for the analysis, one curious question is how do you manage keynames like quotes USD price etc in your code, do you hardcode them, do you have them as environment variables so that you can modify them, some other technique perhaps?
â PirateApp
Jun 26 at 11:14
1
@PirateApp: That depends. If it is one-off code just hard-code them. On the other hand if the possibility exists that you might have some other data source with different structure, you might need to outsource it to a class that gets the data and translates it into some canonical format.
â Graipher
Jun 26 at 11:15
2
Damn, PEP-463 would have made so much Python code sick. There'd also be no need forget
with it too... :O
â Peilonrayz
Jun 26 at 11:25
add a comment |Â
upvoted! thank you for the analysis, one curious question is how do you manage keynames like quotes USD price etc in your code, do you hardcode them, do you have them as environment variables so that you can modify them, some other technique perhaps?
â PirateApp
Jun 26 at 11:14
1
@PirateApp: That depends. If it is one-off code just hard-code them. On the other hand if the possibility exists that you might have some other data source with different structure, you might need to outsource it to a class that gets the data and translates it into some canonical format.
â Graipher
Jun 26 at 11:15
2
Damn, PEP-463 would have made so much Python code sick. There'd also be no need forget
with it too... :O
â Peilonrayz
Jun 26 at 11:25
upvoted! thank you for the analysis, one curious question is how do you manage keynames like quotes USD price etc in your code, do you hardcode them, do you have them as environment variables so that you can modify them, some other technique perhaps?
â PirateApp
Jun 26 at 11:14
upvoted! thank you for the analysis, one curious question is how do you manage keynames like quotes USD price etc in your code, do you hardcode them, do you have them as environment variables so that you can modify them, some other technique perhaps?
â PirateApp
Jun 26 at 11:14
1
1
@PirateApp: That depends. If it is one-off code just hard-code them. On the other hand if the possibility exists that you might have some other data source with different structure, you might need to outsource it to a class that gets the data and translates it into some canonical format.
â Graipher
Jun 26 at 11:15
@PirateApp: That depends. If it is one-off code just hard-code them. On the other hand if the possibility exists that you might have some other data source with different structure, you might need to outsource it to a class that gets the data and translates it into some canonical format.
â Graipher
Jun 26 at 11:15
2
2
Damn, PEP-463 would have made so much Python code sick. There'd also be no need for
get
with it too... :Oâ Peilonrayz
Jun 26 at 11:25
Damn, PEP-463 would have made so much Python code sick. There'd also be no need for
get
with it too... :Oâ Peilonrayz
Jun 26 at 11:25
add a comment |Â
up vote
4
down vote
I previously have written a helper for this, as I've come across it too. If you pass the object and the keys to the object then you can do the same.
def get_nested(obj, keys)
try:
for key in keys:
obj = obj[key]
except KeyError:
return None
return obj
This makes usage:
subset['price'] = get_nested(row, 'quotes USD price'.split())
This allows you to extend the keys argument to a minilanguage for other common methods you'd want to apply to returned objects. One case I had was since JavaScript doesn't like maps much (not objects) some APIs tend to convert dictionaries to arrays, and so I had to search a list of objects for which object I wanted, as I couldn't index the list as I'd get incorrect data at times.
upvoted! thank you for sharing code! i searched for a few questions right here on codereview on flattening nested json and i saw there are a lot of intricacies involved like handling empty keys and any hashable type as opposed to mere string keys, this approach is something i never imagined :)
â PirateApp
Jun 26 at 11:40
add a comment |Â
up vote
4
down vote
I previously have written a helper for this, as I've come across it too. If you pass the object and the keys to the object then you can do the same.
def get_nested(obj, keys)
try:
for key in keys:
obj = obj[key]
except KeyError:
return None
return obj
This makes usage:
subset['price'] = get_nested(row, 'quotes USD price'.split())
This allows you to extend the keys argument to a minilanguage for other common methods you'd want to apply to returned objects. One case I had was since JavaScript doesn't like maps much (not objects) some APIs tend to convert dictionaries to arrays, and so I had to search a list of objects for which object I wanted, as I couldn't index the list as I'd get incorrect data at times.
upvoted! thank you for sharing code! i searched for a few questions right here on codereview on flattening nested json and i saw there are a lot of intricacies involved like handling empty keys and any hashable type as opposed to mere string keys, this approach is something i never imagined :)
â PirateApp
Jun 26 at 11:40
add a comment |Â
up vote
4
down vote
up vote
4
down vote
I previously have written a helper for this, as I've come across it too. If you pass the object and the keys to the object then you can do the same.
def get_nested(obj, keys)
try:
for key in keys:
obj = obj[key]
except KeyError:
return None
return obj
This makes usage:
subset['price'] = get_nested(row, 'quotes USD price'.split())
This allows you to extend the keys argument to a minilanguage for other common methods you'd want to apply to returned objects. One case I had was since JavaScript doesn't like maps much (not objects) some APIs tend to convert dictionaries to arrays, and so I had to search a list of objects for which object I wanted, as I couldn't index the list as I'd get incorrect data at times.
I previously have written a helper for this, as I've come across it too. If you pass the object and the keys to the object then you can do the same.
def get_nested(obj, keys)
try:
for key in keys:
obj = obj[key]
except KeyError:
return None
return obj
This makes usage:
subset['price'] = get_nested(row, 'quotes USD price'.split())
This allows you to extend the keys argument to a minilanguage for other common methods you'd want to apply to returned objects. One case I had was since JavaScript doesn't like maps much (not objects) some APIs tend to convert dictionaries to arrays, and so I had to search a list of objects for which object I wanted, as I couldn't index the list as I'd get incorrect data at times.
answered Jun 26 at 11:36
Peilonrayz
24.3k336101
24.3k336101
upvoted! thank you for sharing code! i searched for a few questions right here on codereview on flattening nested json and i saw there are a lot of intricacies involved like handling empty keys and any hashable type as opposed to mere string keys, this approach is something i never imagined :)
â PirateApp
Jun 26 at 11:40
add a comment |Â
upvoted! thank you for sharing code! i searched for a few questions right here on codereview on flattening nested json and i saw there are a lot of intricacies involved like handling empty keys and any hashable type as opposed to mere string keys, this approach is something i never imagined :)
â PirateApp
Jun 26 at 11:40
upvoted! thank you for sharing code! i searched for a few questions right here on codereview on flattening nested json and i saw there are a lot of intricacies involved like handling empty keys and any hashable type as opposed to mere string keys, this approach is something i never imagined :)
â PirateApp
Jun 26 at 11:40
upvoted! thank you for sharing code! i searched for a few questions right here on codereview on flattening nested json and i saw there are a lot of intricacies involved like handling empty keys and any hashable type as opposed to mere string keys, this approach is something i never imagined :)
â PirateApp
Jun 26 at 11:40
add a comment |Â
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fcodereview.stackexchange.com%2fquestions%2f197241%2faccessing-a-subset-of-keys-from-a-nested-dictionary-in-python%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
2
what do you mean exactly with
The price and volume properties are nested
. please share some example data, and what you expectâ Maarten Fabré
Jun 26 at 7:40
@MaartenFabré thank you for reviewing this question, at the very top I have added a link to the coinmarketcap API containing real data
â PirateApp
Jun 26 at 9:08