Python Requests Returns Incomprehensible Content

I'm trying to parse the site. I don't want to use selenium. Requests is coping. BUT! something strange is happening. I can't cut out the text I need with a regular expression (and it's there - you can see it if you do print(data.text)) But re doesn't see him. If this text is copied to notepad++, it outputs this - it sees these characters as a single line.

import requests
import re

data = requests.get(')

print(data.text)

What is it and how to work with it?pay attention to the line numbers

7

2 Answers

You can try to use their Ajax API to load all usernames + thumb images:

import pandas as pd
import requests

url = '

headers = {'X-Requested-With': 'XMLHttpRequest'}

all_data = []
for p in range(1, 4):  # <-- increase number of pages here
    data = requests.get(url.format(p * 144), headers=headers).json()
    for m in data['models']:
        all_data.append((m['username'], m['display_name'], m['thumb_image'].replace('{ext}', 'jpg')))

df = pd.DataFrame(all_data, columns=['username', 'display_name', 'thumb'])
print(df.head())

Prints:

     username  display_name                                                  thumb
0     wetlilu   Little_Lilu  //
1  mellannie8  mellannieSEX  //
2    mokkoann      mokkoann  //
3    ogurezzi  CynEp-nuCbka  //
4   Pepetka22     _-Katya-_  //

Avoid using . in a regex unless you really want to get any character; here, the usernames (as far as I can see) only contain - and alphanumeric characters, so you can retrieve them with:

re.findall(r'"username":"([\w|-]+)"',data.text)

An even simpler way, which will remove the need to deal with special characters by getting all characters except " is:

re.findall(r'"username":"([^"]+)"',data.text)

So here's a way of getting the info you seek (I joined them into a dictionary, but you can change that to whatever you prefer):

import requests
import re

data = requests.get(')
with open ("return.txt",'w', encoding = 'utf-8') as f:
    f.write(data.text)

names = re.findall(r'"username":"([^"]+)"',data.text)
disp_names = re.findall(r'"display_name":"([^"]+)"',data.text)
thumbs = re.findall(r'"thumb_image":"([^"]+)"',data.text)

names_dict = {name:[disp, thumb.replace('{ext}', 'jpg')] for name, disp, thumb in zip(names, disp_names, thumbs)}

Example

names_dict['JuliaCute']
# ['_Cute',
#  '\\/\\/i.bimbolive.com\\/live\\/055\\/2b0\\/15d\\/xbig_lq\\/d89ef4.jpg']

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

James H. Sterling

James H. Sterling

Environmental Science & Climate Journalist

James Sterling reports on renewable energy developments, climate policy, ecological conservation, and green tech innovations around the globe.

Share this article
Twitter Facebook Pinterest