Accessing a subset of keys from a nested dictionary in Python

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP





.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty margin-bottom:0;







up vote
2
down vote

favorite












I am trying to access data from Coinmarketcap API in Python and came up with the following code:



def fetch_coin_prices(**kwargs):
"""Retrieve cryptocurrency data from CoinMarketCap and return a dictionary
containing coin names with their current prices.

Keyword arguments to this function are mapped to the CoinMarketCap API,
refer to their documentation for their meaning:
https://coinmarketcap.com/api/
"""
response = requests.get(
'https://api.coinmarketcap.com/v2/ticker/',
params=kwargs
)
response.raise_for_status()
coin_data = response.json()

return coin_data.get('data', ).values()


data = fetch_coin_prices(limit=100, start=0, sort='id')


I am trying to write this data to a Redis HMSET that accepts a dictionary and I only want the properties 'id', 'name', 'symbol', 'rank', 'price' and 'volume_24h'. The price and volume properties are nested and therefore I haven't found a way to get everything in 1 go. My redis hmset needs to store data in key value form where key is the coin id and value is the CSV of the current coin.



hmset = 
fieldnames = ['id', 'name', 'symbol', 'rank' ,'price', 'volume_24h']
for row in data:
subset =
subset['id'] = row.get('id', None)
subset['name'] = row.get('name', None)
subset['symbol'] = row.get('symbol', None)
subset['rank'] = row.get('rank', None)
subset['price'] = row.get('quotes', ).get('USD', ).get('price', None)
subset['volume_24h'] = row.get('quotes', ).get('USD', ).get('volume_24h', None)

if all([subset[key] for key in subset]):
csv_string = io.StringIO()
csv_writer = csv.DictWriter(csv_string, fieldnames=fieldnames, extrasaction='ignore' , lineterminator=':')
csv_writer.writerows(data_subset)
hmset[subset['id']] = csv_string.getvalue()


This is what I have come up with so far but it looks very ugly in my opinion to access keys. Is there a better way to do this?







share|improve this question

















  • 2




    what do you mean exactly with The price and volume properties are nested. please share some example data, and what you expect
    – Maarten Fabré
    Jun 26 at 7:40










  • @MaartenFabré thank you for reviewing this question, at the very top I have added a link to the coinmarketcap API containing real data
    – PirateApp
    Jun 26 at 9:08
















up vote
2
down vote

favorite












I am trying to access data from Coinmarketcap API in Python and came up with the following code:



def fetch_coin_prices(**kwargs):
"""Retrieve cryptocurrency data from CoinMarketCap and return a dictionary
containing coin names with their current prices.

Keyword arguments to this function are mapped to the CoinMarketCap API,
refer to their documentation for their meaning:
https://coinmarketcap.com/api/
"""
response = requests.get(
'https://api.coinmarketcap.com/v2/ticker/',
params=kwargs
)
response.raise_for_status()
coin_data = response.json()

return coin_data.get('data', ).values()


data = fetch_coin_prices(limit=100, start=0, sort='id')


I am trying to write this data to a Redis HMSET that accepts a dictionary and I only want the properties 'id', 'name', 'symbol', 'rank', 'price' and 'volume_24h'. The price and volume properties are nested and therefore I haven't found a way to get everything in 1 go. My redis hmset needs to store data in key value form where key is the coin id and value is the CSV of the current coin.



hmset = 
fieldnames = ['id', 'name', 'symbol', 'rank' ,'price', 'volume_24h']
for row in data:
subset =
subset['id'] = row.get('id', None)
subset['name'] = row.get('name', None)
subset['symbol'] = row.get('symbol', None)
subset['rank'] = row.get('rank', None)
subset['price'] = row.get('quotes', ).get('USD', ).get('price', None)
subset['volume_24h'] = row.get('quotes', ).get('USD', ).get('volume_24h', None)

if all([subset[key] for key in subset]):
csv_string = io.StringIO()
csv_writer = csv.DictWriter(csv_string, fieldnames=fieldnames, extrasaction='ignore' , lineterminator=':')
csv_writer.writerows(data_subset)
hmset[subset['id']] = csv_string.getvalue()


This is what I have come up with so far but it looks very ugly in my opinion to access keys. Is there a better way to do this?







share|improve this question

















  • 2




    what do you mean exactly with The price and volume properties are nested. please share some example data, and what you expect
    – Maarten Fabré
    Jun 26 at 7:40










  • @MaartenFabré thank you for reviewing this question, at the very top I have added a link to the coinmarketcap API containing real data
    – PirateApp
    Jun 26 at 9:08












up vote
2
down vote

favorite









up vote
2
down vote

favorite











I am trying to access data from Coinmarketcap API in Python and came up with the following code:



def fetch_coin_prices(**kwargs):
"""Retrieve cryptocurrency data from CoinMarketCap and return a dictionary
containing coin names with their current prices.

Keyword arguments to this function are mapped to the CoinMarketCap API,
refer to their documentation for their meaning:
https://coinmarketcap.com/api/
"""
response = requests.get(
'https://api.coinmarketcap.com/v2/ticker/',
params=kwargs
)
response.raise_for_status()
coin_data = response.json()

return coin_data.get('data', ).values()


data = fetch_coin_prices(limit=100, start=0, sort='id')


I am trying to write this data to a Redis HMSET that accepts a dictionary and I only want the properties 'id', 'name', 'symbol', 'rank', 'price' and 'volume_24h'. The price and volume properties are nested and therefore I haven't found a way to get everything in 1 go. My redis hmset needs to store data in key value form where key is the coin id and value is the CSV of the current coin.



hmset = 
fieldnames = ['id', 'name', 'symbol', 'rank' ,'price', 'volume_24h']
for row in data:
subset =
subset['id'] = row.get('id', None)
subset['name'] = row.get('name', None)
subset['symbol'] = row.get('symbol', None)
subset['rank'] = row.get('rank', None)
subset['price'] = row.get('quotes', ).get('USD', ).get('price', None)
subset['volume_24h'] = row.get('quotes', ).get('USD', ).get('volume_24h', None)

if all([subset[key] for key in subset]):
csv_string = io.StringIO()
csv_writer = csv.DictWriter(csv_string, fieldnames=fieldnames, extrasaction='ignore' , lineterminator=':')
csv_writer.writerows(data_subset)
hmset[subset['id']] = csv_string.getvalue()


This is what I have come up with so far but it looks very ugly in my opinion to access keys. Is there a better way to do this?







share|improve this question













I am trying to access data from Coinmarketcap API in Python and came up with the following code:



def fetch_coin_prices(**kwargs):
"""Retrieve cryptocurrency data from CoinMarketCap and return a dictionary
containing coin names with their current prices.

Keyword arguments to this function are mapped to the CoinMarketCap API,
refer to their documentation for their meaning:
https://coinmarketcap.com/api/
"""
response = requests.get(
'https://api.coinmarketcap.com/v2/ticker/',
params=kwargs
)
response.raise_for_status()
coin_data = response.json()

return coin_data.get('data', ).values()


data = fetch_coin_prices(limit=100, start=0, sort='id')


I am trying to write this data to a Redis HMSET that accepts a dictionary and I only want the properties 'id', 'name', 'symbol', 'rank', 'price' and 'volume_24h'. The price and volume properties are nested and therefore I haven't found a way to get everything in 1 go. My redis hmset needs to store data in key value form where key is the coin id and value is the CSV of the current coin.



hmset = 
fieldnames = ['id', 'name', 'symbol', 'rank' ,'price', 'volume_24h']
for row in data:
subset =
subset['id'] = row.get('id', None)
subset['name'] = row.get('name', None)
subset['symbol'] = row.get('symbol', None)
subset['rank'] = row.get('rank', None)
subset['price'] = row.get('quotes', ).get('USD', ).get('price', None)
subset['volume_24h'] = row.get('quotes', ).get('USD', ).get('volume_24h', None)

if all([subset[key] for key in subset]):
csv_string = io.StringIO()
csv_writer = csv.DictWriter(csv_string, fieldnames=fieldnames, extrasaction='ignore' , lineterminator=':')
csv_writer.writerows(data_subset)
hmset[subset['id']] = csv_string.getvalue()


This is what I have come up with so far but it looks very ugly in my opinion to access keys. Is there a better way to do this?









share|improve this question












share|improve this question




share|improve this question








edited Jun 26 at 2:53









Jamal♦

30.1k11114225




30.1k11114225









asked Jun 26 at 2:39









PirateApp

1316




1316







  • 2




    what do you mean exactly with The price and volume properties are nested. please share some example data, and what you expect
    – Maarten Fabré
    Jun 26 at 7:40










  • @MaartenFabré thank you for reviewing this question, at the very top I have added a link to the coinmarketcap API containing real data
    – PirateApp
    Jun 26 at 9:08












  • 2




    what do you mean exactly with The price and volume properties are nested. please share some example data, and what you expect
    – Maarten Fabré
    Jun 26 at 7:40










  • @MaartenFabré thank you for reviewing this question, at the very top I have added a link to the coinmarketcap API containing real data
    – PirateApp
    Jun 26 at 9:08







2




2




what do you mean exactly with The price and volume properties are nested. please share some example data, and what you expect
– Maarten Fabré
Jun 26 at 7:40




what do you mean exactly with The price and volume properties are nested. please share some example data, and what you expect
– Maarten Fabré
Jun 26 at 7:40












@MaartenFabré thank you for reviewing this question, at the very top I have added a link to the coinmarketcap API containing real data
– PirateApp
Jun 26 at 9:08




@MaartenFabré thank you for reviewing this question, at the very top I have added a link to the coinmarketcap API containing real data
– PirateApp
Jun 26 at 9:08










2 Answers
2






active

oldest

votes

















up vote
2
down vote



accepted










The way using dict.get certainly works, but can become a bit unreadable if you need to chain them.



An alternative is to use exception handling in this case.



subset['volume_24h'] = row.get('quotes', ).get('USD', ).get('volume_24h', None)


is equivalent to



try:
subset['volume_24h'] = row['quotes']['USD']['volume_24h']
except KeyError:
subset['volume_24h'] = None


(This would be a bit neater if PEP-463 had not been rejected:)



subset['volume_24h'] = row['quotes']['USD']['volume_24h'] except KeyError: None





share|improve this answer





















  • upvoted! thank you for the analysis, one curious question is how do you manage keynames like quotes USD price etc in your code, do you hardcode them, do you have them as environment variables so that you can modify them, some other technique perhaps?
    – PirateApp
    Jun 26 at 11:14






  • 1




    @PirateApp: That depends. If it is one-off code just hard-code them. On the other hand if the possibility exists that you might have some other data source with different structure, you might need to outsource it to a class that gets the data and translates it into some canonical format.
    – Graipher
    Jun 26 at 11:15






  • 2




    Damn, PEP-463 would have made so much Python code sick. There'd also be no need for get with it too... :O
    – Peilonrayz
    Jun 26 at 11:25

















up vote
4
down vote













I previously have written a helper for this, as I've come across it too. If you pass the object and the keys to the object then you can do the same.



def get_nested(obj, keys)
try:
for key in keys:
obj = obj[key]
except KeyError:
return None
return obj


This makes usage:



subset['price'] = get_nested(row, 'quotes USD price'.split())


This allows you to extend the keys argument to a minilanguage for other common methods you'd want to apply to returned objects. One case I had was since JavaScript doesn't like maps much (not objects) some APIs tend to convert dictionaries to arrays, and so I had to search a list of objects for which object I wanted, as I couldn't index the list as I'd get incorrect data at times.






share|improve this answer





















  • upvoted! thank you for sharing code! i searched for a few questions right here on codereview on flattening nested json and i saw there are a lot of intricacies involved like handling empty keys and any hashable type as opposed to mere string keys, this approach is something i never imagined :)
    – PirateApp
    Jun 26 at 11:40










Your Answer




StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["\$", "\$"]]);
);
);
, "mathjax-editing");

StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "196"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
convertImagesToLinks: false,
noModals: false,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);








 

draft saved


draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fcodereview.stackexchange.com%2fquestions%2f197241%2faccessing-a-subset-of-keys-from-a-nested-dictionary-in-python%23new-answer', 'question_page');

);

Post as a guest






























2 Answers
2






active

oldest

votes








2 Answers
2






active

oldest

votes









active

oldest

votes






active

oldest

votes








up vote
2
down vote



accepted










The way using dict.get certainly works, but can become a bit unreadable if you need to chain them.



An alternative is to use exception handling in this case.



subset['volume_24h'] = row.get('quotes', ).get('USD', ).get('volume_24h', None)


is equivalent to



try:
subset['volume_24h'] = row['quotes']['USD']['volume_24h']
except KeyError:
subset['volume_24h'] = None


(This would be a bit neater if PEP-463 had not been rejected:)



subset['volume_24h'] = row['quotes']['USD']['volume_24h'] except KeyError: None





share|improve this answer





















  • upvoted! thank you for the analysis, one curious question is how do you manage keynames like quotes USD price etc in your code, do you hardcode them, do you have them as environment variables so that you can modify them, some other technique perhaps?
    – PirateApp
    Jun 26 at 11:14






  • 1




    @PirateApp: That depends. If it is one-off code just hard-code them. On the other hand if the possibility exists that you might have some other data source with different structure, you might need to outsource it to a class that gets the data and translates it into some canonical format.
    – Graipher
    Jun 26 at 11:15






  • 2




    Damn, PEP-463 would have made so much Python code sick. There'd also be no need for get with it too... :O
    – Peilonrayz
    Jun 26 at 11:25














up vote
2
down vote



accepted










The way using dict.get certainly works, but can become a bit unreadable if you need to chain them.



An alternative is to use exception handling in this case.



subset['volume_24h'] = row.get('quotes', ).get('USD', ).get('volume_24h', None)


is equivalent to



try:
subset['volume_24h'] = row['quotes']['USD']['volume_24h']
except KeyError:
subset['volume_24h'] = None


(This would be a bit neater if PEP-463 had not been rejected:)



subset['volume_24h'] = row['quotes']['USD']['volume_24h'] except KeyError: None





share|improve this answer





















  • upvoted! thank you for the analysis, one curious question is how do you manage keynames like quotes USD price etc in your code, do you hardcode them, do you have them as environment variables so that you can modify them, some other technique perhaps?
    – PirateApp
    Jun 26 at 11:14






  • 1




    @PirateApp: That depends. If it is one-off code just hard-code them. On the other hand if the possibility exists that you might have some other data source with different structure, you might need to outsource it to a class that gets the data and translates it into some canonical format.
    – Graipher
    Jun 26 at 11:15






  • 2




    Damn, PEP-463 would have made so much Python code sick. There'd also be no need for get with it too... :O
    – Peilonrayz
    Jun 26 at 11:25












up vote
2
down vote



accepted







up vote
2
down vote



accepted






The way using dict.get certainly works, but can become a bit unreadable if you need to chain them.



An alternative is to use exception handling in this case.



subset['volume_24h'] = row.get('quotes', ).get('USD', ).get('volume_24h', None)


is equivalent to



try:
subset['volume_24h'] = row['quotes']['USD']['volume_24h']
except KeyError:
subset['volume_24h'] = None


(This would be a bit neater if PEP-463 had not been rejected:)



subset['volume_24h'] = row['quotes']['USD']['volume_24h'] except KeyError: None





share|improve this answer













The way using dict.get certainly works, but can become a bit unreadable if you need to chain them.



An alternative is to use exception handling in this case.



subset['volume_24h'] = row.get('quotes', ).get('USD', ).get('volume_24h', None)


is equivalent to



try:
subset['volume_24h'] = row['quotes']['USD']['volume_24h']
except KeyError:
subset['volume_24h'] = None


(This would be a bit neater if PEP-463 had not been rejected:)



subset['volume_24h'] = row['quotes']['USD']['volume_24h'] except KeyError: None






share|improve this answer













share|improve this answer



share|improve this answer











answered Jun 26 at 10:50









Graipher

20.4k42981




20.4k42981











  • upvoted! thank you for the analysis, one curious question is how do you manage keynames like quotes USD price etc in your code, do you hardcode them, do you have them as environment variables so that you can modify them, some other technique perhaps?
    – PirateApp
    Jun 26 at 11:14






  • 1




    @PirateApp: That depends. If it is one-off code just hard-code them. On the other hand if the possibility exists that you might have some other data source with different structure, you might need to outsource it to a class that gets the data and translates it into some canonical format.
    – Graipher
    Jun 26 at 11:15






  • 2




    Damn, PEP-463 would have made so much Python code sick. There'd also be no need for get with it too... :O
    – Peilonrayz
    Jun 26 at 11:25
















  • upvoted! thank you for the analysis, one curious question is how do you manage keynames like quotes USD price etc in your code, do you hardcode them, do you have them as environment variables so that you can modify them, some other technique perhaps?
    – PirateApp
    Jun 26 at 11:14






  • 1




    @PirateApp: That depends. If it is one-off code just hard-code them. On the other hand if the possibility exists that you might have some other data source with different structure, you might need to outsource it to a class that gets the data and translates it into some canonical format.
    – Graipher
    Jun 26 at 11:15






  • 2




    Damn, PEP-463 would have made so much Python code sick. There'd also be no need for get with it too... :O
    – Peilonrayz
    Jun 26 at 11:25















upvoted! thank you for the analysis, one curious question is how do you manage keynames like quotes USD price etc in your code, do you hardcode them, do you have them as environment variables so that you can modify them, some other technique perhaps?
– PirateApp
Jun 26 at 11:14




upvoted! thank you for the analysis, one curious question is how do you manage keynames like quotes USD price etc in your code, do you hardcode them, do you have them as environment variables so that you can modify them, some other technique perhaps?
– PirateApp
Jun 26 at 11:14




1




1




@PirateApp: That depends. If it is one-off code just hard-code them. On the other hand if the possibility exists that you might have some other data source with different structure, you might need to outsource it to a class that gets the data and translates it into some canonical format.
– Graipher
Jun 26 at 11:15




@PirateApp: That depends. If it is one-off code just hard-code them. On the other hand if the possibility exists that you might have some other data source with different structure, you might need to outsource it to a class that gets the data and translates it into some canonical format.
– Graipher
Jun 26 at 11:15




2




2




Damn, PEP-463 would have made so much Python code sick. There'd also be no need for get with it too... :O
– Peilonrayz
Jun 26 at 11:25




Damn, PEP-463 would have made so much Python code sick. There'd also be no need for get with it too... :O
– Peilonrayz
Jun 26 at 11:25












up vote
4
down vote













I previously have written a helper for this, as I've come across it too. If you pass the object and the keys to the object then you can do the same.



def get_nested(obj, keys)
try:
for key in keys:
obj = obj[key]
except KeyError:
return None
return obj


This makes usage:



subset['price'] = get_nested(row, 'quotes USD price'.split())


This allows you to extend the keys argument to a minilanguage for other common methods you'd want to apply to returned objects. One case I had was since JavaScript doesn't like maps much (not objects) some APIs tend to convert dictionaries to arrays, and so I had to search a list of objects for which object I wanted, as I couldn't index the list as I'd get incorrect data at times.






share|improve this answer





















  • upvoted! thank you for sharing code! i searched for a few questions right here on codereview on flattening nested json and i saw there are a lot of intricacies involved like handling empty keys and any hashable type as opposed to mere string keys, this approach is something i never imagined :)
    – PirateApp
    Jun 26 at 11:40














up vote
4
down vote













I previously have written a helper for this, as I've come across it too. If you pass the object and the keys to the object then you can do the same.



def get_nested(obj, keys)
try:
for key in keys:
obj = obj[key]
except KeyError:
return None
return obj


This makes usage:



subset['price'] = get_nested(row, 'quotes USD price'.split())


This allows you to extend the keys argument to a minilanguage for other common methods you'd want to apply to returned objects. One case I had was since JavaScript doesn't like maps much (not objects) some APIs tend to convert dictionaries to arrays, and so I had to search a list of objects for which object I wanted, as I couldn't index the list as I'd get incorrect data at times.






share|improve this answer





















  • upvoted! thank you for sharing code! i searched for a few questions right here on codereview on flattening nested json and i saw there are a lot of intricacies involved like handling empty keys and any hashable type as opposed to mere string keys, this approach is something i never imagined :)
    – PirateApp
    Jun 26 at 11:40












up vote
4
down vote










up vote
4
down vote









I previously have written a helper for this, as I've come across it too. If you pass the object and the keys to the object then you can do the same.



def get_nested(obj, keys)
try:
for key in keys:
obj = obj[key]
except KeyError:
return None
return obj


This makes usage:



subset['price'] = get_nested(row, 'quotes USD price'.split())


This allows you to extend the keys argument to a minilanguage for other common methods you'd want to apply to returned objects. One case I had was since JavaScript doesn't like maps much (not objects) some APIs tend to convert dictionaries to arrays, and so I had to search a list of objects for which object I wanted, as I couldn't index the list as I'd get incorrect data at times.






share|improve this answer













I previously have written a helper for this, as I've come across it too. If you pass the object and the keys to the object then you can do the same.



def get_nested(obj, keys)
try:
for key in keys:
obj = obj[key]
except KeyError:
return None
return obj


This makes usage:



subset['price'] = get_nested(row, 'quotes USD price'.split())


This allows you to extend the keys argument to a minilanguage for other common methods you'd want to apply to returned objects. One case I had was since JavaScript doesn't like maps much (not objects) some APIs tend to convert dictionaries to arrays, and so I had to search a list of objects for which object I wanted, as I couldn't index the list as I'd get incorrect data at times.







share|improve this answer













share|improve this answer



share|improve this answer











answered Jun 26 at 11:36









Peilonrayz

24.3k336101




24.3k336101











  • upvoted! thank you for sharing code! i searched for a few questions right here on codereview on flattening nested json and i saw there are a lot of intricacies involved like handling empty keys and any hashable type as opposed to mere string keys, this approach is something i never imagined :)
    – PirateApp
    Jun 26 at 11:40
















  • upvoted! thank you for sharing code! i searched for a few questions right here on codereview on flattening nested json and i saw there are a lot of intricacies involved like handling empty keys and any hashable type as opposed to mere string keys, this approach is something i never imagined :)
    – PirateApp
    Jun 26 at 11:40















upvoted! thank you for sharing code! i searched for a few questions right here on codereview on flattening nested json and i saw there are a lot of intricacies involved like handling empty keys and any hashable type as opposed to mere string keys, this approach is something i never imagined :)
– PirateApp
Jun 26 at 11:40




upvoted! thank you for sharing code! i searched for a few questions right here on codereview on flattening nested json and i saw there are a lot of intricacies involved like handling empty keys and any hashable type as opposed to mere string keys, this approach is something i never imagined :)
– PirateApp
Jun 26 at 11:40












 

draft saved


draft discarded


























 


draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fcodereview.stackexchange.com%2fquestions%2f197241%2faccessing-a-subset-of-keys-from-a-nested-dictionary-in-python%23new-answer', 'question_page');

);

Post as a guest













































































Popular posts from this blog

Greedy Best First Search implementation in Rust

Function to Return a JSON Like Objects Using VBA Collections and Arrays

C++11 CLH Lock Implementation