it's once again time for Me Asking Noob-Level Questions About jq
15
Replies
·
0
Boosts
(Specifically, I have a big ol' JSON dataset with like tens of thousands of objects containing a bunch of key-value pairs. However it is missing a crucial key-value. I need to be like, "ok, for every object combine value1+value2+value3, number each unique combination so I'm dealing with a nice identifier number rather than a monster string, and add newKey: uniqueID to each object." I feel this would be easy if I were used to 1) programming and 2) JSON but I'm not. I could do this in a GUI spreadsheet program but my laptop only has so much RAM.)
edit: I fired up the gaming rig and did it by hand in a spreadsheet program with a lot of concatenation and vlookup, but I am still interested in finding a way to do it more automatically.
(The "gaming rig" is a second-hand computer from Free Geek that has, like, a discrete graphics card and more than 8 GB of RAM, which by my standards is basically a supercomputer.)
@nev there's a trick here: you can do it in two passes, spending extra I/O to save on RAM.
In the first pass you output every value1+value2+value3, then you sort+uniq that list and output a concatenated-value → unique_combination mapping table into a file.
Then run a second jq program which starts by reads this mapping table into a variable (--rawfile varname filename), and iterate over your original file adding a concatenated-value→newKey mapping to each entry.
Does that make sense?
@gnomon this does make sense and is pretty much what I did but in spreadsheet form with UNIQUE, COUNTUNIQUE, and some filling down
@nev threw this together for you, lightly tested but you can probably edit it to do what you want https://paste.sr.ht/~d6/38c9166a97d2ce2b198bd2ac18c8435de3a6591d
@d6 thank you so much! i'll give it a try
@nev now I'm like do you know anyone hosting OpenOffice or whatever on a cloud. oh you could burn googles compute on it with Google sheets
@t54r4n1 that's what I usually do. I may have to fire up the Gaming Rig which has more RAM to do it comfortably. However it would be nice if I could figure out how to do it in a terminal.
