Jq stuff: Difference between revisions

From SoftwareGuy
Jump to navigation Jump to search
Mark (talk | contribs)
No edit summary
Mark (talk | contribs)
 
(13 intermediate revisions by the same user not shown)
Line 1: Line 1:
Making csv output from a json input, example.
== Using jq ==


Staring with the follow sample json:
[[Parsing]]


<pre>
Command to return the id number of the last scan:
{"avg":16383,"detectorStatus":[{"sumBPort":"0","isAvailable":"true","temp":"33.0","isHvAdequate":"true","hvSetPoint":"681.0","threshold":"-22000","id":"0_0_0","hvEnabled":"true","iobPort":"0","hv":"656.0"},{"sumBPort":"1","isAvailable":"true","temp":"33.75","isHvAdequate":"true","hvSetPoint":"684.0","threshold":"-22000","id":"1_0_1","hvEnabled":"true","iobPort":"0","hv":"660.0"},{"sumBPort":"2","isAvailable":"true","temp":"33.875","isHvAdequate":"true","hvSetPoint":"684.0","threshold":"-22000","id":"2_0_2","hvEnabled":"true","iobPort":"0","hv":"660.0"},{"sumBPort":"3","isAvailable":"true","temp":"34.25","isHvAdequate":"true","hvSetPoint":"699.0","threshold":"-22000","id":"3_0_3","hvEnabled":"true","iobPort":"0","hv":"676.0"},{"sumBPort":"4","isAvailable":"true","temp":"35.0","isHvAdequate":"true","hvSetPoint":"693.0","threshold":"-22000","id":"4_0_4","hvEnabled":"true","iobPort":"0","hv":"668.0"},{"sumBPort":"5","isAvailable":"false","id":"5_0_5","iobPort":"0"},{"sumBPort":"6","isAvailable":"false","id":"6_0_6","iobPort":"0"},{"sumBPort":"7","isAvailable":"false","id":"7_0_7","iobPort":"0"},{"sumBPort":"8","isAvailable":"false","id":"8_0_8","iobPort":"0"},{"sumBPort":"9","isAvailable":"false","id":"9_0_9","iobPort":"0"},{"sumBPort":"10","isAvailable":"false","id":"10_0_10","iobPort":"0"},{"sumBPort":"11","isAvailable":"false","id":"11_0_11","iobPort":"0"},{"sumBPort":"0","isAvailable":"true","temp":"34.0","isHvAdequate":"true","hvSetPoint":"708.0","threshold":"-22000","id":"12_1_0","hvEnabled":"true","iobPort":"1","hv":"684.0"},{"sumBPort":"1","isAvailable":"true","temp":"34.0","isHvAdequate":"true","hvSetPoint":"726.0","threshold":"-22000","id":"13_1_1","hvEnabled":"true","iobPort":"1","hv":"699.0"},{"sumBPort":"2","isAvailable":"true","temp":"34.5","isHvAdequate":"true","hvSetPoint":"738.0","threshold":"-22000","id":"14_1_2","hvEnabled":"true","iobPort":"1","hv":"715.0"},{"sumBPort":"3","isAvailable":"true","temp":"35.375","isHvAdequate":"true","hvSetPoint":"750.0","threshold":"-22000","id":"15_1_3","hvEnabled":"true","iobPort":"1","hv":"730.0"},{"sumBPort":"4","isAvailable":"true","temp":"36.0","isHvAdequate":"true","hvSetPoint":"765.0","threshold":"-22000","id":"16_1_4","hvEnabled":"true","iobPort":"1","hv":"742.0"},{"sumBPort":"5","isAvailable":"false","id":"17_1_5","iobPort":"1"},{"sumBPort":"6","isAvailable":"false","id":"18_1_6","iobPort":"1"},{"sumBPort":"7","isAvailable":"false","id":"19_1_7","iobPort":"1"},{"sumBPort":"8","isAvailable":"false","id":"20_1_8","iobPort":"1"},{"sumBPort":"9","isAvailable":"false","id":"21_1_9","iobPort":"1"},{"sumBPort":"10","isAvailable":"false","id":"22_1_10","iobPort":"1"},{"sumBPort":"11","isAvailable":"false","id":"23_1_11","iobPort":"1"},{"sumBPort":"0","isAvailable":"true","temp":"30.875","isHvAdequate":"true","hvSetPoint":"775.0","threshold":"-1","id":"24_2_0","hvEnabled":"true","iobPort":"2","hv":"699.0"},{"sumBPort":"1","isAvailable":"true","temp":"27.875","isHvAdequate":"true","hvSetPoint":"830.0","threshold":"-1","id":"25_2_1","hvEnabled":"true","iobPort":"2","hv":"758.0"},{"sumBPort":"2","isAvailable":"true","temp":"26.875","isHvAdequate":"true","hvSetPoint":"830.0","threshold":"-22000","id":"26_2_2","hvEnabled":"true","iobPort":"2","hv":"754.0"},{"sumBPort":"3","isAvailable":"true","temp":"26.875","isHvAdequate":"true","hvSetPoint":"802.0","threshold":"-1","id":"27_2_3","hvEnabled":"true","iobPort":"2","hv":"777.0"},{"sumBPort":"4","isAvailable":"true","temp":"25.875","isHvAdequate":"true","hvSetPoint":"767.0","threshold":"-1","id":"28_2_4","hvEnabled":"true","iobPort":"2","hv":"734.0"},{"sumBPort":"5","isAvailable":"true","temp":"27.875","isHvAdequate":"true","hvSetPoint":"767.0","threshold":"-1","id":"29_2_5","hvEnabled":"true","iobPort":"2","hv":"691.0"},{"sumBPort":"6","isAvailable":"false","id":"30_2_6","iobPort":"2"},{"sumBPort":"7","isAvailable":"false","id":"31_2_7","iobPort":"2"},{"sumBPort":"8","isAvailable":"false","id":"32_2_8","iobPort":"2"},{"sumBPort":"9","isAvailable":"false","id":"33_2_9","iobPort":"2"},{"sumBPort":"10","isAvailable":"false","id":"34_2_10","iobPort":"2"},{"sumBPort":"11","isAvailable":"false","id":"35_2_11","iobPort":"2"},{"sumBPort":"0","isAvailable":"true","temp":"30.875","isHvAdequate":"true","hvSetPoint":"855.0","threshold":"-1","id":"36_3_0","hvEnabled":"true","iobPort":"3","hv":"793.0"},{"sumBPort":"1","isAvailable":"true","temp":"26.875","isHvAdequate":"true","hvSetPoint":"910.0","threshold":"-5000","id":"37_3_1","hvEnabled":"true","iobPort":"3","hv":"855.0"},{"sumBPort":"2","isAvailable":"true","temp":"26.875","isHvAdequate":"true","hvSetPoint":"885.0","threshold":"-22000","id":"38_3_2","hvEnabled":"true","iobPort":"3","hv":"855.0"},{"sumBPort":"3","isAvailable":"true","temp":"24.875","isHvAdequate":"true","hvSetPoint":"860.0","threshold":"-1","id":"39_3_3","hvEnabled":"true","iobPort":"3","hv":"828.0"},{"sumBPort":"4","isAvailable":"true","temp":"24.875","isHvAdequate":"true","hvSetPoint":"849.0","threshold":"-1","id":"40_3_4","hvEnabled":"true","iobPort":"3","hv":"781.0"},{"sumBPort":"5","isAvailable":"true","temp":"27.875","isHvAdequate":"true","hvSetPoint":"860.0","threshold":"-1","id":"41_3_5","hvEnabled":"true","iobPort":"3","hv":"797.0"},{"sumBPort":"6","isAvailable":"false","id":"42_3_6","iobPort":"3"},{"sumBPort":"7","isAvailable":"false","id":"43_3_7","iobPort":"3"},{"sumBPort":"8","isAvailable":"false","id":"44_3_8","iobPort":"3"},{"sumBPort":"9","isAvailable":"false","id":"45_3_9","iobPort":"3"},{"sumBPort":"10","isAvailable":"false","id":"46_3_10","iobPort":"3"},{"sumBPort":"11","isAvailable":"false","id":"47_3_11","iobPort":"3"}],"determinationMade":false,"duration":22.461,"highSpeedViolation":false,"isSuspicious":false,"lowSpeedViolation":false,"max":-32768,"min":65535,"mph":1.8530931,"pb":0,"pixelsPerRaster":1000,"rastersPerImage":0,"renderData":{"claheCellHeightDiv":75,"claheCellWidthDiv":72,"claheClipLimit":1.2,"claheOn":true,"colorMap":0,"editVersion":0,"gaussianHeight":5,"gaussianWidth":5,"histoHigh":255,"histoLow":0,"impulseRemovalOn":false,"mConfig":{"claheCellHeightDiv":75,"claheCellWidthDiv":72,"claheClipLimit":1.2,"claheOn":true,"colorMap":0,"gaussianHeight":5,"gaussianWidth":5,"histoHigh":255,"histoLow":0,"impulseRemovalOn":false,"sharpenValue":0.5,"uniformSampleOn":false},"pixelCount":0,"scaleFactor":2.4707909,"sharpenValue":0.5,"type":"uvx","uniformSampleOn":false},"scanId":"26","systemSerialNumber":"fa4c5a84-3ab8-403d-a1b9-18a2351bcdfa","timestamp":"Thu Mar 16 21:05:37 GMT+00:00 2023","version":1.2}
</pre>Using the following command will generate the desired csv output:


<code>jq -r '.detectorStatus[] | [.id, .hvSetPoint] | @csv' 26.json > 26.csv</code>
<code>cnt=$(jq '.lastVehicleRecord.scans | length' heartbeat)</code>


Resulting in the following data:
Fancy way to find the scanId of a selected system via bash and jq


"0_0_0","681.0"
<pre>if [ -z $1 ]; then whichIE="uvx"; else whichIE=$1; fi
printf ".lastVehicleRecord.scans[] | select(.type | endswith(\""$whichIE"\"))\n" > parms.jq


"1_0_1","684.0"
wget -qO heartbeat <nowiki>http://$ip/heartbeat</nowiki>
cnt=$(jq '.lastVehicleRecord.scans | length' heartbeat)


"2_0_2","684.0"
id=$(jq -f parms.jq heartbeat | jq '.id')
if [ ! -z $id ]; then
    wget -qO $id.orig.tiff <nowiki>http://$ip/requestImage/?scanId=$id</nowiki>
fi</pre>More interesting jq examples:


"3_0_3","699.0"
<code>"plateCoordinates":[898, 210, 144, 44] "plateCoordinatesRelative":[898, 210, 144, 44],</code>


"4_0_4","693.0"
<code>jq ".plateCoordinates | .[1]" upload_test/2023-05-12_12:08:07.txt</code>


"5_0_5",
== Tiff info parsing ==
Parsing examples of .tiff files extracing meta data from the headers.


"6_0_6",
<code>tiffinfo 2023-05-07\ 13_57_17_evx_left_3845.tiff | grep 65001 | sed 's/  Tag 65001: //' | sed 's/\\//'g | jq</code>


"7_0_7",
<code>tiffinfo 2023-05-07\ 13_57_17_evx_left_3845.tiff | grep 65001 | sed 's/  Tag 65001: //' | sed 's/\\//'g | jq '.timestamp'</code>


"8_0_8",
<code>tiffinfo 2023-05-07\ 13_57_17_evx_left_3845.tiff | grep 65001 | sed 's/  Tag 65001: //' | sed 's/\\//'g | jq '.renderData.type'</code>


"9_0_9",
Remove stderr from stream


"10_0_10",
<code>tiffinfo 189.tiff 2>/dev/null | grep 65001 | sed 's/  Tag 65001: //' | sed 's/\\//'g | jq '.renderData.type'</code>


"11_0_11",
== LPR parsing (Axis P1455) ==
The camera should be set to "save full frame".


"12_1_0","708.0"
Parsing the json from the axis camera:


"13_1_1","726.0"
<code>jq ".ImageArray[0].BinaryImage" /home4/Events/upload_test/2023-06-15_15:27:01.txt > image.b64</code>


"14_1_2","738.0"
Next remove the quotes in the image.b64 file, or:


"15_1_3","750.0"
<code>jq --raw-output ".ImageArray[0].BinaryImage" /home4/Events/upload_test/2023-06-15_15:27:01.txt > image.b64</code>


"16_1_4","765.0"
Then:


"17_1_5",
<code>base64 -d image.b64 > image.jpg</code>


"18_1_6",
Now to get the lpr image:


"19_1_7",
<code>jq ".imagesURI[0]"  /home4/Events/upload_test/2023-06-15_15:27:01.txt</code>


"20_1_8",
then use that url to pull the license plate image.  You may have to trim the returned url to just include "<code>getImage&name=3</code>", not the full url...


"21_1_9",
Alternatively you could just grab the plate coordinates from the json and clip the image, presumably.


"22_1_10",
<code>jq ".plateCoordinates"  /home4/Events/upload_test/2023-06-15_15:27:01.txt</code>


"23_1_11",
== Service Log parsing ==
As of July 25, 2023, the services logs have issues with quotes and backslashes.


"24_2_0","775.0"
Basic parsing of the log is done like this:


"25_2_1","830.0"
<code>jq ".[0].message" getServiceLogs.json</code>


"26_2_2","830.0"
provides just the first message element.  


"27_2_3","802.0"
I edit the file and remove the quote around the contents of the "message" element.  (in emacs macro search for message, space forward to the starting quote, now search for }" then search for dateLong and end macro.  Next I am removing the /, both of these in emacs.


"28_2_4","767.0"
A bunch of sed stuff to help with the parsing...


"29_2_5","767.0"
<code>sed 's/\\\"\\\"/""""/g' getServiceLogs.json | sed 's/\\//g' | sed 's/\}"/\}/g' | sed 's/""/"/g' | sed 's/\"\[{/\[{/g' | sed 's/}\]\"/}\]/g' | sed 's/\"{/{/g' > getServiceLogs2.json</code>
 
"30_2_6",
 
"31_2_7",
 
"32_2_8",
 
"33_2_9",
 
"34_2_10",
 
"35_2_11",
 
"36_3_0","855.0"
 
"37_3_1","910.0"
 
"38_3_2","885.0"
 
"39_3_3","860.0"
 
"40_3_4","849.0"
 
"41_3_5","860.0"
 
"42_3_6",
 
"43_3_7",
 
"44_3_8",
 
"45_3_9",
 
"46_3_10",
 
"47_3_11",

Latest revision as of 15:55, 26 July 2023

Using jq[edit | edit source]

Parsing

Command to return the id number of the last scan:

cnt=$(jq '.lastVehicleRecord.scans | length' heartbeat)

Fancy way to find the scanId of a selected system via bash and jq

if [ -z $1 ]; then whichIE="uvx"; else whichIE=$1; fi
printf ".lastVehicleRecord.scans[] | select(.type | endswith(\""$whichIE"\"))\n" > parms.jq

wget -qO heartbeat http://$ip/heartbeat
cnt=$(jq '.lastVehicleRecord.scans | length' heartbeat)

id=$(jq -f parms.jq heartbeat | jq '.id')
if [ ! -z $id ]; then
    wget -qO $id.orig.tiff http://$ip/requestImage/?scanId=$id
fi

More interesting jq examples:

"plateCoordinates":[898, 210, 144, 44] "plateCoordinatesRelative":[898, 210, 144, 44],

jq ".plateCoordinates | .[1]" upload_test/2023-05-12_12:08:07.txt

Tiff info parsing[edit | edit source]

Parsing examples of .tiff files extracing meta data from the headers.

tiffinfo 2023-05-07\ 13_57_17_evx_left_3845.tiff | grep 65001 | sed 's/  Tag 65001: //' | sed 's/\\//'g | jq

tiffinfo 2023-05-07\ 13_57_17_evx_left_3845.tiff | grep 65001 | sed 's/  Tag 65001: //' | sed 's/\\//'g | jq '.timestamp'

tiffinfo 2023-05-07\ 13_57_17_evx_left_3845.tiff | grep 65001 | sed 's/  Tag 65001: //' | sed 's/\\//'g | jq '.renderData.type'

Remove stderr from stream

tiffinfo 189.tiff 2>/dev/null | grep 65001 | sed 's/  Tag 65001: //' | sed 's/\\//'g | jq '.renderData.type'

LPR parsing (Axis P1455)[edit | edit source]

The camera should be set to "save full frame".

Parsing the json from the axis camera:

jq ".ImageArray[0].BinaryImage" /home4/Events/upload_test/2023-06-15_15:27:01.txt > image.b64

Next remove the quotes in the image.b64 file, or:

jq --raw-output ".ImageArray[0].BinaryImage" /home4/Events/upload_test/2023-06-15_15:27:01.txt > image.b64

Then:

base64 -d image.b64 > image.jpg

Now to get the lpr image:

jq ".imagesURI[0]"  /home4/Events/upload_test/2023-06-15_15:27:01.txt

then use that url to pull the license plate image. You may have to trim the returned url to just include "getImage&name=3", not the full url...

Alternatively you could just grab the plate coordinates from the json and clip the image, presumably.

jq ".plateCoordinates"  /home4/Events/upload_test/2023-06-15_15:27:01.txt

Service Log parsing[edit | edit source]

As of July 25, 2023, the services logs have issues with quotes and backslashes.

Basic parsing of the log is done like this:

jq ".[0].message" getServiceLogs.json

provides just the first message element.

I edit the file and remove the quote around the contents of the "message" element. (in emacs macro search for message, space forward to the starting quote, now search for }" then search for dateLong and end macro. Next I am removing the /, both of these in emacs.

A bunch of sed stuff to help with the parsing...

sed 's/\\\"\\\"/""""/g' getServiceLogs.json | sed 's/\\//g' | sed 's/\}"/\}/g' | sed 's/""/"/g' | sed 's/\"\[{/\[{/g' | sed 's/}\]\"/}\]/g' | sed 's/\"{/{/g' > getServiceLogs2.json