How Can I Replace or Remove Html Entities Like " " Using Beautifulsoup 4
I Am Processing Html Using Python and the Beautifulsoup 4 Library and I Can't Find an Obvious Way to Replace with a Space. Instead It Seems to Be Converted to...
I am processing HTML using Python and the BeautifulSoup 4 library and I can't find an obvious way to replace with a space. Instead it seems to be converted to a Unicode non-breaking space character.
Am I missing something obvious? What is the best way to replace with a normal space using BeautifulSoup?
Edit to add that I am using the latest version, BeautifulSoup 4, so the convertEntities=BeautifulSoup.HTML_ENTITIES option in Beautiful Soup 3 isn't available.
5 Answers
>>> soup = BeautifulSoup('<div>a b</div>')
>>> soup.prettify(formatter=lambda s: s.replace(u'\xa0', ' '))
u'<html>\n <body>\n <div>\n a b\n </div>\n </body>\n</html>'
See Entities in the documentation. BeautifulSoup 4 produces proper Unicode for all entities:
An incoming HTML or XML entity is always converted into the corresponding Unicode character.
Yes, is turned into a non-breaking space character. If you really want those to be space characters instead, you'll have to do a unicode replace.
You can simply replace the non-breaking space unicode with a normal space.
nonBreakSpace = u'\xa0'
soup = soup.replace(nonBreakSpace, ' ')
A benefit is that even though you are using BeautifulSoup, you do not need to.
I had issues with json that soup.prettify() did not fix, so it worked with unicodedata.normalize():
import unicodedata
soup = BeautifulSoup(r.text, 'html.parser')
dat = soup.find('span', attrs={'class': 'date'})
print(f"date prints fine:'{dat.text}'")
print(f"json:{json.dumps(dat.text)}")
mydate = unicodedata.normalize("NFKD",dat.text)
print(f"json after normalizing:'{json.dumps(mydate)}'")
date prints fine:'03 Nov 19 17:51'
json:"03\u00a0Nov\u00a019\u00a017:51"
json after normalizing:'"03 Nov 19 17:51"'
Admittedly this is not using BeautifulSoup, but a more straightforward solution today may be some combination of html.unescape and unicodedata.normalize, depending on your data and what you want to do exactly.
>>> from html import unescape
>>> s = unescape('An enthusiastic member of the community')# Using the import here
>>> print(s)
>>> 'An enthusiastic member of the\xa0community'
>>> import unicodedata
>>> s = unicodedata.normalize('NFKC', s)
>>> print(s)
>>> 'An enthusiastic member of the community'
Marcus Vance is a cybersecurity auditor and technology writer dedicated to educating the public about online safety, data privacy regulations, enterprise security, and emerging cyber threats.
