Discussion:
Stripping curly quotes and other chars from outgoing Mail
(too old to reply)
Roger Howard
2006-05-10 18:30:32 UTC
Permalink
My sister is encountering a lot of negative feedback directed at her and her Mac-using office. They send out press releases and other communications a dozen times a day to a wide variety of media outlets. Some percentage of the recipients are complaining of nasty "?" characters appearing frequently throughout the body of her text.

As I suspected initially, I've been able to trace this back to use of characters like curly quotes, em dashes, and other extended characters. Some number of the recipients are on systems where they aren't correctly displaying these characters.

Since she can live without these chars my question is whether anyone knows of a service, script, or other easy tool for her to simplify or strip out such characters from emails, either while composing or outbound from Mail.app? I can whip together an AppleScript to do it, but if there's an easier (or existing) tool that'd be ideal.

FYI, she doesn't want to just select Plain Text, so that's not an option (if it would help). We need to allow Rich Text, but just convert or strip characters - for instance, em dashes should become hyphens, curly quotes become straight quotes, etc.

Ideas?

Thanks,
Roger
Charles Dyer
2006-05-10 19:06:24 UTC
Permalink
Post by Roger Howard
My sister is encountering a lot of negative feedback directed at
her and her Mac-using office. They send out press releases and
other communications a dozen times a day to a wide variety of media
outlets. Some percentage of the recipients are complaining of nasty
"?" characters appearing frequently throughout the body of her text.
As I suspected initially, I've been able to trace this back to use
of characters like curly quotes, em dashes, and other extended
characters. Some number of the recipients are on systems where they
aren't correctly displaying these characters.
Since she can live without these chars my question is whether
anyone knows of a service, script, or other easy tool for her to
simplify or strip out such characters from emails, either while
composing or outbound from Mail.app? I can whip together an
AppleScript to do it, but if there's an easier (or existing) tool
that'd be ideal.
FYI, she doesn't want to just select Plain Text, so that's not an
option (if it would help). We need to allow Rich Text, but just
convert or strip characters - for instance, em dashes should become
hyphens, curly quotes become straight quotes, etc.
Is she using MS Word? If so, it's easily fixed: The AutoCorrect item
in the Tools menu opens a dialog which has four panes. #1 is
AutoCorrect itself; this automatically replaces (r) with ®, (c) with
©, ... with …, etc. The second pane, AutoFormat As You Type, replaces
straight quotes with curly quotes, automatically generates
superscripts for items like 1st, 2nd, makes bulleted lists, etc.
AutoText is for boilerplate. AutoFormat sets specific items from the
first two panes.

If she's not using Word, which app is she using? That might have an
easy fix, too.
Roger Howard
2006-05-10 20:04:15 UTC
Permalink
Post by Charles Dyer
If she's not using Word, which app is she using? That might have an
easy fix, too.
Thanks Charles.

Basically these are non-technical users - of course. They copy/paste
content from a variety of sources - Word docs, PDFs, and "printable"
versions of Web pages - into Mail.app, pretty it up further
(inserting attachments, etc), and then send. So we need to somehow
clean the content in their outgoing mails. I looked into WordService,
but it seems this would require selecting several filters in
succession, probably not a good solution for them.

What about options to create a new service? I could whip up a filter
in any common scripting language... or?

It's of course a problem with the receiving mail client, but
nevertheless they're blaming my sis, and we can't have that! And
since these characters aren't critical (no one cares if it's an em-
dash or a hyphen) it'd be ideal to solve it on the composing end.

Cheers -R
steve harley
2006-05-10 20:09:40 UTC
Permalink
What about options to create a new service? I could whip up a filter in
any common scripting language... or?
TextWeilder?

but i'm not sure how it would interact with RTF ...
Mark Smith
2006-05-10 20:18:18 UTC
Permalink
Post by Charles Dyer
Is she using MS Word?
If I'm reading Roger's post right, she's using Mail.app.

no idea if it suits your sister's purpose, but one way that should
work would be to compose in markdown syntax and pass the input
through e.g. markdown.pl and smartypant.pl. If you then pass the
resulting HTML to a Mail template with appropriate header info...

...a bit more than you'd expect Joe Public to jump through to get
an email out the door, but you like this kind of thing, right ?

mark.
Karl Kuehn
2006-05-10 20:30:44 UTC
Permalink
Post by Mark Smith
Post by Charles Dyer
Is she using MS Word?
If I'm reading Roger's post right, she's using Mail.app.
But Mail.app does not usually do any of these fancy things... that
is more of a Word thing, so I agree that copy-pasting in from Word is
probably where these problematic characters are coming from.

The only solution is probably going to be to write up an AppleScript
that would manually be run on every email before it was sent (it
would probably look for the foremost Mail.app window, and then act on
the text in that) that would go and replace a list of problematic
characters. This is not going to be a nice fool-proof solution, but
the only other solutions I can think of are either to force plain-
text (at least once in the process), or to do this on the server
side, and that is going to get very messy and assumes that you have
access to that.

--
Karl Kuehn
***@softhome.net
Roger Howard
2006-05-10 21:13:02 UTC
Permalink
Post by Karl Kuehn
Post by Mark Smith
Post by Charles Dyer
Is she using MS Word?
If I'm reading Roger's post right, she's using Mail.app.
But Mail.app does not usually do any of these fancy things... that
is more of a Word thing, so I agree that copy-pasting in from Word
is probably where these problematic characters are coming from.
It's not just from Word... she copies from any source... many of the
problem cases were from text copied from Safari, so let's not fixate
on Word as the source of the problem, or a chokepoint where it would
make sense to implement a solution (if it was always copying from
Word I'd just make a macro that cleaned the text before putting it on
the clipboard). And no, I'm not blaming the problem on Mail.app
either, as it seems to handle these characters fine -- it's obviously
a problem with the recipient's mail client, but well that just
doesn't matter... it's gotta be solved on the sending end.
Post by Karl Kuehn
The only solution is probably going to be to write up an
AppleScript that would manually be run on every email before it was
sent (it would probably look for the foremost Mail.app window, and
then act on the text in that)
Which unfortunately seems beyond the realm of Mail's bizarre
AppleScript object model... for instance, no getting the contents of
the frontmost window, no selection property of window objects, the
selection property of the application obj does not return text but
only selected messages, etc...
Post by Karl Kuehn
that would go and replace a list of problematic characters. This is
not going to be a nice fool-proof solution, but the only other
solutions I can think of are either to force plain-text (at least
once in the process), or to do this on the server side, and that is
going to get very messy and assumes that you have access to that.
Unfortunately I've been butting my head up against Mail's AppleScript
implementation (while I've avoided Mail scripting like the plague,
I've been AppleScripting for nearly 10 years) so until I find a
workaround there (if possible at all) that doesn't require her to
jump through some big hoops, it doesn't seem doable that way. Am
thinking I'm going to have to resort to a custom Service as at least
those have access to selected text in Mail.app
steve harley
2006-05-10 22:29:50 UTC
Permalink
Am thinking I'm going to
have to resort to a custom Service as at least those have access to
selected text in Mail.app
in addition to TextWielder, i'd look at On My Command; since it's
a contextual menu, it can work with selected text (but unlike a
Service it can't replace the text without clobbering the clipboard)

<http://free.abracode.com/cmworkshop/on_my_command.html>

also, On My Command commands can be plugged into QuickSilver
steve harley
2006-05-10 21:08:38 UTC
Permalink
Post by Karl Kuehn
But Mail.app does not usually do any of these fancy things... that
is more of a Word thing, so I agree that copy-pasting in from Word is
probably where these problematic characters are coming from.
Word, XPress, InDesign, etc. may automate such fancy things, but
any text app, including Mail, lets you do them yourself; they
could also be pasted in from typographically articulate web
pages; and not everyone who uses them by instinct knows to avoid
them in email, especially since for years they appeared to work
fine, even in "plain text", in Mac to Mac email

the question (though not the immediate solution) may be one of
Unicode compatibility -- is Mail maintaining the proper Unicode
code points for these glyphs or using "high ASCII" instead (i
don't know how this works with RTF)? and is the recipient's
client Unicode aware?
Mark Smith
2006-05-10 21:04:33 UTC
Permalink
Post by Karl Kuehn
Post by Mark Smith
Post by Charles Dyer
Is she using MS Word?
If I'm reading Roger's post right, she's using Mail.app.
But Mail.app does not usually do any of these fancy
things...
it can if you get excited in rich text mode but...
Post by Karl Kuehn
that is more of a Word thing, so I agree that
copy-pasting in from Word is probably where these
problematic characters are coming from.
...this is likely and Roger has commented that its probably coming
from a variety of sources.
Post by Karl Kuehn
The only solution is probably going to be to write up an
AppleScript that would manually be run on every email before
it was sent (it would probably look for the foremost
Mail.app window, and then act on the text in that) that
would go and replace a list of problematic characters. This
is not going to be a nice fool-proof solution, but the only
other solutions I can think of are either to force plain-
text (at least once in the process)...
...which might be preferable.

mark.
Chad Leigh
2006-05-10 21:42:36 UTC
Permalink
Post by Roger Howard
FYI, she doesn't want to just select Plain Text, so that's not an
option (if it would help). We need to allow Rich Text, but just
convert or strip characters - for instance, em dashes should become
hyphens, curly quotes become straight quotes, etc.
Ideas?
Can she generate PDF files to email out?

Chad
Roger Howard
2006-05-10 21:52:41 UTC
Permalink
Post by Chad Leigh
Post by Roger Howard
FYI, she doesn't want to just select Plain Text, so that's not an
option (if it would help). We need to allow Rich Text, but just
convert or strip characters - for instance, em dashes should
become hyphens, curly quotes become straight quotes, etc.
Ideas?
Can she generate PDF files to email out?
Nope...

-R
Google Kreme
2006-05-11 00:06:33 UTC
Permalink
On 10 May 2006, at 12:30 , Roger Howard wrote: My sister is
encountering a lot of negative feedback directed at her and her Mac-
using office. They send out press releases and other communications a
dozen times a day to a wide variety of media outlets. Some percentage
of the recipients are complaining of nasty "?" characters appearing
frequently throughout the body of her text.

As I suspected initially, I've been able to trace this back to use of
characters like curly quotes, em dashes, and other extended
characters. Some number of the recipients are on systems where they
aren't correctly displaying these characters.

Since she can live without these chars my question is whether anyone
knows of a service, script, or other easy tool for her to simplify or
strip out such characters from emails, either while composing or
outbound from Mail.app? I can whip together an AppleScript to do it,
but if there's an easier (or existing) tool that'd be ideal.

Erm... is she sending attachments? If not, just set mail to plain
text, no?

FYI, she doesn't want to just select Plain Text, so that's not an
option (if it would help). We need to allow Rich Text, but just
convert or strip characters - for instance, em dashes should become
hyphens, curly quotes become straight quotes, etc.

Well, that is odd. Rich text, but not quite as rich as other text?

Well, there's always WordService, it seems to do everything (Services
- Format - Reformat, or Command-Shift-7). Are services scriptable? If
so, you could have them run a script before they send out these mails.

Or, you could tell them to ignore the whining from the out of date
windows clients who's mail programs can't handle text-encoding. But
that would be MY solution, and so unlikely to be workable for the non-
asshole :)

Or, if you have control of the mailserver you could run some sort of
filter on outgoing mail that "cleaned" the email for the
retards^WWindows 98 users on the distribution list.

OK, wait, here is what I would recommend:

Have them paste all their text into Text Wrangler, run clean text
(Convert to ASCII, straighten quotes, zap gremlins--all scriptable),
then paste it into mail, THEN add their RTF kruft.

Text-Wrangler is even freeware.

--
"You're an elf and you're going to wear panties like an elf." David
Sedaris, Santaland Diaries
Roger Howard
2006-05-11 00:30:24 UTC
Permalink
Post by Google Kreme
Erm... is she sending attachments? If not, just set mail to plain
text, no?
No, they like rich text, and they do use attachments and images.
Post by Google Kreme
Well, that is odd. Rich text, but not quite as rich as other text?
Rich text doesn't seem to the be the problem, and Plain Text isn't a
solution. We've tried. Setting to plain text doesn't change curly
quotes to straight quotes, or em-dashes to hyphens, so I fail to see
how this helps?
Post by Google Kreme
Well, there's always WordService, it seems to do everything
(Services - Format - Reformat, or Command-Shift-7). Are services
scriptable? If so, you could have them run a script before they
send out these mails.
I've considered WordService - very nice, but it looks like what
they'd need to do would require multiple passes with different
WordService filters. Not something we want to train a monkey to do...
Post by Google Kreme
Or, you could tell them to ignore the whining from the out of date
windows clients who's mail programs can't handle text-encoding. But
that would be MY solution, and so unlikely to be workable for the
non-asshole :)
Oh, I'm an asshole, don't get me wrong! If this were just me, I
wouldn't have even asked for this advice. But she's in the
communications business; they have to reach as many people as
possible, and create as little annoyance as possible while doing it.
She's got a CEO who now thinks this is a "Mac problem", which is a
veiled way of saying "figure it out or get a PC like everyone else in
the office".
Post by Google Kreme
Or, if you have control of the mailserver you could run some sort
of filter on outgoing mail that "cleaned" the email for the
retards^WWindows 98 users on the distribution list.
I wish... no, they don't even know what a mail server is.
Post by Google Kreme
Have them paste all their text into Text Wrangler, run clean text
(Convert to ASCII, straighten quotes, zap gremlins--all
scriptable), then paste it into mail, THEN add their RTF kruft
Hehehe, ok, well that's what I'd do too... but I can see you don't
deal much with these kind of people :)

Thanks guys, I think we can close this thread... I've got enough
options, but it basically comes down to some annoying choices. In all
likelihood I'm just going to send her my superfab "Clean the Cruft"
script which just processes whatever's on the clipboard. She can
copy, run the script, then paste. Will even add GUI Scripting support
for Mail.app so if its the frontmost app it'll do the copy and paste
parts for her... Boom, done. In the end, seems the easiest way.

-R
Google Kreme
2006-05-11 00:52:40 UTC
Permalink
Post by Roger Howard
Post by Google Kreme
Or, you could tell them to ignore the whining from the out of date
windows clients who's mail programs can't handle text-encoding.
But that would be MY solution, and so unlikely to be workable for
the non-asshole :)
Oh, I'm an asshole, don't get me wrong!
Oh, I KNOW THAT! i was just giving your sister the benefit of the
doubt.
Post by Roger Howard
Post by Google Kreme
Have them paste all their text into Text Wrangler, run clean text
(Convert to ASCII, straighten quotes, zap gremlins--all
scriptable), then paste it into mail, THEN add their RTF kruft
Hehehe, ok, well that's what I'd do too... but I can see you don't
deal much with these kind of people :)
Ah, if only! Fortuenately, I can almost always get away with
"Windows is retarded, buy a Mac." Not that they do, but at least
they know where the problem is.

Word Service, I think, deserves a look. Its Command-keyed Reformat
Text is pretty darn good. I would not be surprised if it straightened
“” and ‘’ characters.

<tries it on preceding paragraph>

... or not. Hrm, I am a tad surprised.

OK, write a service then. Really, it should be dead simple, once you
figure out the initial 'how do I write a service?' part. :D
--
"You're an elf and you're going to wear panties like an elf." David
Sedaris, Santaland Diaries
steve harley
2006-05-11 02:39:27 UTC
Permalink
Post by Google Kreme
OK, write a service then. Really, it should be dead simple, once you
figure out the initial 'how do I write a service?' part. :D
that's why i recommended TextWielder

<http://www.commongrnd.com/~blenkush/products.html>
John Maliga
2006-05-11 23:47:11 UTC
Permalink
Post by Roger Howard
Thanks guys, I think we can close this thread... I've got enough
options, but it basically comes down to some annoying choices.
Ooops. Sorry, but I meant to add TextSoap to the mix. Works with its
"scrubbers" (including built-in straighten quotes) on selected text
through the contextual menu. You can also create a customized
"scrubber". And now, apparently, works with Services in Mail.app.

http://www.unmarked.com/textsoap/index.html

--JM

Loading...