Remove lines from multiline json


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Remove lines from multiline json
# 1  
Old 03-07-2018
Remove lines from multiline json

Hi ,
When extracting the data from API end point ,its giving multi line json .I want to remove certain lines with
group": "tag" or tget and respect "item" values [only want to keep group": "url and along with rest of values.Any idea how to remove

Code:
python test.py /data{" Id":" 7554317""group":"get", "item":"xx5e1"],"fields":[" Id","type"," Parameters","xx"]}


 {
      "time": 1520460953,
      " Parameters1": [
        {
          "group": "tag",
          "item": "else"
        },
        {
          "group": "url",
          "item": "https://xxx.con"
        },
        {
          "group": "tget",
          "item": "a73927xxxxx"
        }
      ],
      " ite": " 1877",
      "type": "impression",
      "userId": "da52yjnh"
    },
    {
      "time": 1520460953,
      "Parameters1": [
        {
          "group": "tag",
          "item": "else"
        },
        {
          "group": "url",
          "item": "https://xxx1.con"
        },
        {
          "group": "tget",
          "item": "a73927fayyy"
        }
      ],
      " ite": "9222 ",
      "type": "impression",
      "userId": " jxbccyqsl"
    },

# 2  
Old 03-08-2018
Hi akil

You request is very difficult to understand and ambiguous at best. Do not be surprise if the solutions are not close to what you wish.

It would had helped if you would had shown the desired output. Since the manipulation of the json posted would have to be handled by the python script test.py, it would had helped, as well, to see what it has.

Here's some points for whatever is worth to you, since the posted json is a non-valid fragment.
Example of deleting entries you do not want.

Code:
import json

some_json_string = '''
{
	"something": [{
			"time": 1520460953,
			"Parameters1": [{
					"group": "tag",
					"item": "else"
				},
				{
					"group": "url",
					"item": "https://xxx.con"
				},
				{
					"group": "tget",
					"item": "a73927xxxxx"
				}
			],
			" ite": " 1877",
			"type": "impression",
			"userId": "da52yjnh"
		},
		{
			"time": 1520460953,
			"Parameters1": [{
					"group": "tag",
					"item": "else"
				},
				{
					"group": "url",
					"item": "https://xxx1.con"
				},
				{
					"group": "tget",
					"item": "a73927fayyy"
				}
			],
			" ite": "9222 ",
			"type": "impression",
			"userId": " jxbccyqsl"
		}
	]
}
'''

data = json.loads(some_json_string)
print("ORIGINAL json")
print(some_json_string)

for a in data['something']:
    a['Parameters1'] = a['Parameters1'][1]

new_some_json_string = json.dumps(data, indent=2)
print("MODIFIED json")
print(new_some_json_string)

Output:

Code:
ORIGINAL json

{
	"something": [{
			"time": 1520460953,
			"Parameters1": [{
					"group": "tag",
					"item": "else"
				},
				{
					"group": "url",
					"item": "https://xxx.con"
				},
				{
					"group": "tget",
					"item": "a73927xxxxx"
				}
			],
			" ite": " 1877",
			"type": "impression",
			"userId": "da52yjnh"
		},
		{
			"time": 1520460953,
			"Parameters1": [{
					"group": "tag",
					"item": "else"
				},
				{
					"group": "url",
					"item": "https://xxx1.con"
                }
				},
				{
					"group": "tget",
					"item": "a73927fayyy"
				}
			],
			" ite": "9222 ",
			"type": "impression",
			"userId": " jxbccyqsl"
		}
	]
}

MODIFIED json
{
  "something": [
    {
      "time": 1520460953,
      "Parameters1": {
        "group": "url",
        "item": "https://xxx.con"
      },
      " ite": " 1877",
      "type": "impression",
      "userId": "da52yjnh"
    },
    {
      "time": 1520460953,
      "Parameters1": {
        "group": "url",
        "item": "https://xxx1.con"
      },
      " ite": "9222 ",
      "type": "impression",
      "userId": " jxbccyqsl"
    }
  ]
}

Of course, this is JUST an example where I have to make the json valid and correct even the Parameters1 key since in the posted portion one of them has a leading space and the other doesn't.
# 3  
Old 03-08-2018
When trying to execute ,getting the following error message
Code:
import json
data = json.loads("/xx/sites_extract_1.json")
print("ORIGINAL json")
print(some_json_string)

for a in data['something']:
    a['Parameters1'] = a['Parameters1'][1]

new_some_json_string = json.dumps(data, indent=2)
print("MODIFIED json")
print(new_some_json_string)

Traceback (most recent call last):
File "test1.py", line 2, in <module>
data = json.loads("/xx/sites_extract_1.json")
File "/usr/lib/python2.7/json/__init__.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

---------- Post updated at 01:06 AM ---------- Previous update was at 01:01 AM ----------

expecting output
Code:
"time": 1520460953,
      " Parameters1": [
  
        {
          "group": "url",
          "item": "https://xxx.con"
        }
   
      ],
      " ite": " 1877",
      "type": "impression",
      "userId": "da52yjnh"
    },

# 4  
Old 03-08-2018
Quote:
Originally Posted by akil
[...]
Traceback (most recent call last):
File "test1.py", line 2, in <module>
data = json.loads("/xx/sites_extract_1.json")
File "/usr/lib/python2.7/json/__init__.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Essentially, it is telling you that it is no valid json.
I gave you an example of how to load a json string. You are passing it what it appears to be a file path.
Maybe:
Code:
with open('/xx/sites_extract_1.json') as data_file:   
    data = json.load(data_file)


Last edited by Aia; 03-08-2018 at 02:31 AM..
# 5  
Old 03-08-2018
Thanks Aia,working fine
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove first meta key from json records using shell

Hi All, I need to get rid of initial meta key from json files with enclosed parenthesis from start and end of the lines which has total 4000 lines. here is the sample Json records : {"start": true, "meta": {"name": "xyz", "creation": "2017-07-14T16:20:06.000+02:00"}} I need to remove... (7 Replies)
Discussion started by: Cloud_Ninja
7 Replies

2. Shell Programming and Scripting

awk to remove lines that do not start with digit and combine line or lines

I have been searching and trying to come up with an awk that will perform the following on a converted text file (original is a pdf). 1. Since the first two lines are (begin with) text they are removed 2. if $1 is a number then all text is merged (combined) into one line until the next... (3 Replies)
Discussion started by: cmccabe
3 Replies

3. Shell Programming and Scripting

How to remove multiline HTML tags from a file?

I am trying to remove a multiline HTML tag and its contents from a few HTML files following the same basic pattern. So far using regex and sed have been unsuccessful. The HTML has a basic structure like this (with the normal HTML stuff around it): <div id="div1"> <div class="div2"> <other... (4 Replies)
Discussion started by: threesixtyfive
4 Replies

4. Shell Programming and Scripting

Remove lines that are subsets of other lines in File

Hello everyone, Although it seems easy, I've been stuck with this problem for a moment now and I can't figure out a way to get it done. My problem is the following: I have a file where each line is a sequence of IP addresses, example : 10.0.0.1 10.0.0.2 10.0.0.5 10.0.0.1 10.0.0.2... (5 Replies)
Discussion started by: MisterJellyBean
5 Replies

5. Shell Programming and Scripting

Two files, remove lines from second based on lines in first

I have two files, a keepout.txt and a database.csv. They're unsorted, but could be sorted. keepout: user1 buser3 anuser19 notheruser27 database: user1,2343,"information about",field,blah,34 user2,4231,"mo info",etc,stuff,43 notheruser27,4344,"hiya",thing,more thing,423... (4 Replies)
Discussion started by: esoffron
4 Replies

6. Shell Programming and Scripting

Remove multiline text between brackets

I have some text in a file like so This is {the first day of} my life. What I would like as output is This is my life. Any text between the curly braces is removed. In the forums I've found statements like sed 's/<*>//g' but the problem is that I think that... (12 Replies)
Discussion started by: climatron
12 Replies

7. UNIX for Dummies Questions & Answers

Want to remove all lines but not latest 50 lines from a file

Hi, I have a huge file which has Lacs of lines. File system got full. I want your guys help to suggest me a solution so that I can remove all lines from that file but not last 50,000 lines. I want solution which can remove lines from existing file so that I can have some space left with. (28 Replies)
Discussion started by: prashant2507198
28 Replies

8. Shell Programming and Scripting

remove blank lines and merge lines in shell

Hi, I'm not a expert in shell programming, so i've come here to take help from u gurus. I'm trying to tailor a csv file that i got to make it work for the LOAD FROM command. I've a datatable csv of the below format - --in file format xx,xx,xx ,xx , , , , ,,xx, xxxx,, ,, xxx,... (11 Replies)
Discussion started by: dvah
11 Replies

9. Shell Programming and Scripting

Remove all lines except lines starting with [

Hello, I am trying to remove all the lines in file except lines starting with [ How can i accomplish this? Thank you very much in advance. (4 Replies)
Discussion started by: maxo
4 Replies

10. Shell Programming and Scripting

Remove All Lines Between Two Unique Lines

Hi all! Im wondering if its possible to remove all lines between two lines. Im working with a document like this: data1 data2 <Remove> data3 data4 </Remove> data5 data6 I need it to end up like this if that possible: data1 data2 data5 data6 There are multiple instances of... (2 Replies)
Discussion started by: Grizzly
2 Replies
Login or Register to Ask a Question