Dan Fixes Dot Blog

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
.====== Decrapify and Send ======

It's finally time to plug the ereader into the computer!

When you first plug in the ereader, Windows will make a bunch of noises.  It might complain that there's a problem with the drive you just plugged in - ignore it, it's lying, the drive is fine, do not accept its offer of help.  Wait for Windows to calm down.  Once it's stopped pinging at you, Calibre should have added another couple of buttons to its top toolbar; next to "Library," sixth from the left, will be "Device" and "Card A."  If you click on either of those two buttons, it'll show the books in the main memory or SD card of the ereader.  Have a look if you like, there might be stuff in there already or there might not, and then go back to Library.

Now that we're done with the initial setup, the process for getting and reading ebooks goes like this:
  - Download a book to your computer's hard drive;
  - Bring the book into Calibre;e
  - Use Calibre to decrapify the book;
  - Send the decrapified book to your ereader and find a comfy chair.



===== Getting books into Calibre =====

In Calibre, click on "Add Books," the green icon in the top left.  A file chooser window appears.  Navigate to wherever you downloaded your ebook files to, probably your Downloads folder.  Highlight the book or books you want to add (holding down the Shift key is handy for selecting many books at once) and click Open or press Enter.

A window will pop up briefly saying something like "Reading metadata and adding books to library," it'll probably be gone pretty quick, don't worry if you miss it.  The book you just added will be visible in Calibre, with the title all the way on the left, followed by the author.  Stuff like author, title, publisher etc is known as **metadata** inside Calibre.

This metadata might be messed up.  The author might be swapped with the title, or it might not have a cover, or it might not have any information about the series, or something like that - if so (and if you care), read the next bit, if not then skip it and go straight to Decrapifying ePubs.



==== If a book's metadata is messed up (and if you care enough to fix it) ====

Your ereader doesn't show you filenames, it only shows you title and author, so it's useful to get those right.  If the book is part of a series and you fill in the Series information, then it'll be added to the Collections page of your ereader's menu, which can be nice and helpful.

To change any of this data, right-click on the book in Calibre's list, choose "Edit Metadata," and then "Edit metadata individually," and there'll be options where you can fill out any information you like, or even download information and alternate covers from the internet.

=== Quick fixes to the most common metadata screwups ===

  * **Author and Title reversed** - Sometimes you'll download a book called something like "J.R.R. Tolkien" by The Lord of The Rings.  The title and author sometimes get swapped round by mistake.  This happens often enough that there's a dedicated "Swap Author and Title" button which can be found to the left of the Title field, it looks like a blue box and a green box doing a dance.
  * **Author listed last-name-first** - Sometimes an author will be listed by library sort order rather than normal-looking order (for example, a Terry Pratchett book might have "Pratchett, Terry" as the author name).  Author Name and Author Sort are different things and Calibre treats them quite rightly as different things.  Fill out the Author(s) field first-name-first, and then hit the rightward-facing green arrow next to the Author(s) field, and the sort-order will get entered and set to "last name, first name" automatically.
  * **I have a book by R.L. Stine and another book by R. L. Stine and Calibre thinks these are different guys** - people are very inconsistent with how many spaces they put in multi-abbreviated author names like R.L. Stine or C. J. Cherryh, and this isn't a problem until computers get involved because computers are famously pedantic.  Calibre has helpful features like author autocomplete (start typing "r." and Calibre autosuggests the rest of the author name so you can just hit Down Arrow then Enter rather than typing the whole thing every time) but helpful features like this depend on you being consistent with author names.  Decide whether it's "R.L. Stine" without the space or "R. L. Stine" with the space - it doesn't matter which, so long as you make a decision and commit to sticking with it.  Then highlight all the affected disorganized Stine books and use the "Edit Metadata In Bulk" option to make all the appropriate author names the same.  Then figure out what to do about Robert Heinlein and Robert A. Heinlein and Robert Anson Heinlein I guess.
  * **I don't have a cover (or my cover sucks and I want one from a different edition)** - Just to the right of the cover image is a submenu headed "Change cover" with some buttons beneath.  You can try the "Download cover" option which will automatically search Google Images and Amazon and some other places, and then choose the cover you want.  This depends on your Title and Author being set correctly.  If you don't see any you like, you can search in your web browser for a new cover image, save it to your hard drive, and then select it using the "Browse" button.

=== Title and Author are really all you need ===

The only metadata that really needs to be correct is the book's title and the author.  If you want to be meticulous about things like setting the publisher and the edition number and the date of publication then that's fine, a lot of people take delight in curating these things and making their library neat and tidy.  Your ereader will still read the book either way, so fill out as much information as brings you joy.  If you're the meticulous sort and you already have a folder full of a thousand epubs, avoid the temptation to import them all at once; bringing them into Calibre one or two authors at a time can help keep this fun rather than overwhelming.

=== Series information is not super-necessary but nice to have ===

The PRS-505 has a "Collections" menu that can show books by series, and this can be handy.  For an example from my library, I've got a twelve-volume collection of the complete short stories of Theodore Sturgeon; if I were to leave volume 2 named "The complete stories of Theodore Sturgeon volume II: Microcosmic God" then when I browsed my ebooks by title, the ereader would truncate these massive titles to something like "The complete stories of Theodor..." repeated twelve times, and that's not ideal because I can't tell which one's which.

Instead, I set the Title to "Microcosmic God" and set the Series information to "Complete Stories of Theodore Sturgeon [2]" and then I can see all the titles laid out neatly in order in the "Complete Stories of Theodore Sturgeon" menu inside Collections on the ereader, and that's much nicer.

But, again, the ereader will still read the book either way, so this is absolutely not a big deal.



===== Decrapifying ePubs =====

From here on out, we assume that your ebooks are in .epub format (unless you wanna read comics/manga, see later on for a special section on that).

Old ereaders choke on new ebook files because new ebook files are full of unnecessarily massive images and font files that old ereaders don't have enough memory to figure out.  Calibre can fix all those problems and give you an .epub that will work reliably and load fast.  Loading fast is important because the CPU takes so much more power than the screen - this can make the difference between going days between charges, or weeks!

Here's what we'll do:

  - (always) Click the "Resize Cover" button, to make the book cover an appropriate size for our ereader's screen.  One click.
  - (usually) Click the "Polish Books" button, to shrink down the font files by removing the letters that the book doesn't actually use.  Some publishers like to do things like use huge ornate capitals for the first letter of each chapter, but they might use five or ten of the letters in a font that has all the letters plus numbers plus accented characters and emoji and stuff, so we're going to remove the characters that aren't used in the book, and sometimes this can shrink a book's filesize to a tenth of what it was before.  This process takes a minute but is automatic, and can be started with one or two clicks.
  - (sometimes) Shrink down the images inside the book to fit our ereader's screen.  Ebooks as downloaded are often full of images that are four or five times bigger than even a new ereader's screen, in case you ever want to zoom in on an author's nostril hairs (and also so that Amazon can sell you a new Kindle with more memory).  When we resize them to an appropriate size, they'll use far less memory, load much faster, and look much better on our screen.  This takes a few more clicks, but it's not too bad.



==== Step 1: Resize the cover ====

Highlight the book in Calibre that you want to convert.  The cover image will appear in the right-hand panel, and sometimes you'll also see some tags and possibly the blurb from the back cover.  In the bottom-right corner of the cover image will be a bit of text showing the dimensions of the cover - probably a couple thousand pixels tall and over a thousand wide.

Go up to the menu bar at the top of the screen, with all the massive buttons.  Click on "Resize Cover."  Notice that the text on the bottom right of the cover image is now 768 pixels tall (or less), and 600 pixels wide (or less).  That's it, we're done.

=== Step 1 in bulk ===
If you want to resize the cover on a whole bunch of books at once, you can totally do that - hold down Shift and click to select many books, then hit Resize Cover just like before, and it'll do them all at once.



==== Step 2: shrink massive goofy font files and compress images ====

Polishing a book file will fix those goofy massive font files and compress the images inside the book.  Note that compressing is, confusingly, not the same as resizing, that'll happen in the next step.  Compressing means reducing the image quality, but not in a way that you'll actually be able to tell on an e-ink screen.

In the main Calibre window, highlight the book you're working on and click "Polish books" in the big top bar full of icons (looks like a dusting-brush).

If you're curious about what each thing does, you can click the "About" text next to the option.  Here are the options to check:
  * CHECK: Embed all referenced fonts
  * CHECK: Subset all embedded fonts
  * Unchecked: Smarten punctuation
  * CHECK: Update metadata in the book files
  * CHECK: Update the cover in the book files
  * Unchecked: Add/replace metadata as a "book jacket" page
  * Unchecked: Remove a previously inserted book jacket
  * CHECK: Remove unused CSS rules from the book
  * CHECK: Losslessly compress images
  * Unchecked: Download external resources
  * Unchecked: Add soft hyphens
  * Unchecked: Remove soft hyphens
  * Unchecked: Upgrade book internals
These options are saved, so you only have to do the checking/unchecking once.

Below this list is "Show report."  If you have this checked, then once the polishing is done, it'll pop up a new window showing what the polishing accomplished - check it if you're curious, have it unchecked if you're doing lots of books at once and want to save some time.

Click on OK.  The book polishing window closes, leaving the main Calibre window open - at the very bottom-right, the "Jobs" indicator starts spinning, showing that Calibre is working on stuff.  If you like, you can click on Jobs to see how it's getting on.  This might take a minute.  Once it's done, if you had "Show report" checked then a report will appear showing what it did.

Once it's finished, you might well be done, let's check.  Right-click on the book in Calibre, select Open book folder -> Open book folder (or press O on your keyboard) and this'll open Calibre's folder for this ebook in your file manager.  You'll see two ebook files; one will have an .epub extension, and the other will have an .original_epub extension.  If the .epub one is like a couple of megs or less, then it's almost certainly gonna load on just about anything - if not, then proceed to the next step.

=== Step 2 in bulk ===
Got a load of epubs that all need this treatment?  Hold down the Shift key to select a whole bunch at once and then proceed as above, then click on Jobs to see how long it's gonna take and whether you've got time to make a cup of tea.



==== Step 3: resize the images inside the book (if any) ====

I was putting some Goosebumps books on my kid's ereader, and noticed that they were like twelve megabytes each and it took ages to turn pages.  An .epub that's just text should be like 100 or 200kb for a kid-length story, and these books aren't illustrated, so what the hell?

Turns out, it was because each chapter heading was an image containing the chapter number in a spooky font, and each one of those images was so enormous that you could print it out as a poster and hang it on your wall.  If we make those images the same width as the ereader's screen, then not only do we save a massive amount of space on the memory card, we save the ereader the work of loading a too-big image and then resizing it to fit the screen as we read the book, so that saves a lot of battery life too.

Right-click the book you're working on and select "Edit Book."  The ebook editor window appears.

Here you can see the guts of your epub.  Don't worry about the guts.  Look at the top of the screen with the File, Edit, Tools etc menus, and find Plugins.  Click on Plugins, then click Bulk Image Resizer - or, press CTRL + Alt + Shift + R on your keyboard.

The Bulk Image Resizer plugin isn't as user-friendly as the Resize Cover plugin, and doesn't save settings.  Where it asks for a maximum resolution applied to the shorter side of images, enter 600.  Leave the following quality setting at 85 (or knock it down a bit if you want to save even more battery and memory card space, you can go down to about 70 before you notice any difference).  Under encoding type, choose JPEG.  Click OK.

A new window appears, showing the differences that the Bulk Image Resizer plugin made to the book, with the original on the left and the changes on the right.  You might see some HTML code, you might see some images, you can scroll through looking at the before-and-after if you're curious but it doesn't really matter.  In the bottom right of this window are buttons labelled "Revert changes" and "Close" - click on Close, and the window closes, taking you back to the ebook editor window.

Now, save the book (File -> Save, or third icon from the left in the top icon bar, looks like a floppy disk).

You can now close the ebook editor window, and we're done with this step.

=== Step 3 in bulk ===
Step 3 can't be done in bulk at time of writing (spring 2025), you'll have to do each file one at a time, sorry.  One of these days I'll write a Calibre plugin that'll do all three steps in one click and can be run in bulk, but that day is not today.


**We have now decrapified our book!**


===== Sending books to the ereader =====
Plug in the ereader, give it a sec to let Calibre detect it, highlight the book or books to send to the reader, right-click and select Send to device -> Send to storage card A.

**Don't try to send your entire library at once!**  See "Notes on SD Card Corruption" further down this page.  Send maybe twenty books at a time.

This will take a minute.  Check the progress by clicking Jobs in the very bottom right of the Calibre window.

Once it's done, give it another ten seconds or so, and then (this is important) click the downward-facing arrow to the right of "Device" in the top big-icon menu, and select "Eject this device."

Once ejected, you can unplug the reader.  Give it a minute to spin its arrows (give it a LONG time if you've just put dozens of books on it) and you're done, go read some books!

===== Notes on SD card corruption =====
I've had a couple of SD cards get wonky and quit accepting or reading new books.  I'm still narrowing down what, exactly, causes this to happen, but I can quite reliably make it happen by copying either **too much data** or **too many books** to the ereader in a single operation.  **Update 2025-05-15:** <del>New information has been brought to light</del> old information has been dug up from a swamp of fifteen-year-old forum posts and also I've spent //far// too much time experimenting and being frustrated, see "SD card too big" below.

=== How do you know when it's happened? ===
Here are the indicators/symptoms of SD card corruption that I've noticed:
  * Recently added books are displayed as filenames (often with numbers on the end), rather than book titles, when you know that their metadata was correct in Calibre;
  * "Collections" menu goes missing or displays 0 Collections, when there were collections present before;
  * "No book" shown under "Continue Reading" heading;
  * "No bookmark" shown under the Bookmarks heading (assuming you had set bookmarks before);
  * "Unable to communicate with device" errors in Calibre when plugging the ereader back into the computer;
  * Books appearing with one or zero pages, or appearing normally but giving an error when you try to read them (although this particular one might not be indicative of SD card corruption but just one bad book).

=== How it happens (working theory) ===
After you're done adding books to the ereader and you've ejected and unplugged it from your machine, the ereader will sit and spin its arrow-circle for a minute or two.  During this time, the ereader is having a good sniff around its filesystem, finding books that it doesn't yet know about, querying each new file to get its title and author and collection data to show in the menus, making some preliminary decisions about how each book should be displayed, and writing all this data to the following files:
  SD card/Sony Reader/database/cache.xml
  SD card/Sony Reader/database/cacheExt.xml
If this process goes wrong while the ereader is writing to the cache.xml or cacheExt.xml files (or maybe if the process goes wrong while those files are open for writing) then the ereader will never send the "Okay I'm done writing here" bits to the filesystem, and one or both of these files will end up with a chunk missing off the end, which makes the whole file no good and can often cause further problems elsewhere on the SD card.  Tragically, our reading positions are also stored in these files, so the ereader will forget which page we were on!  We can reliably cause this SD card corruption by sending too many books in one go, sending a few enormous books in one go, or (added 2025-05-15) trying to use an SD card bigger than 4 gigabytes, sigh.

== Too many books in one go ==
Through experimentation I've figured out that starting with an empty SD card and adding several hundred (small, like under a megabyte each) books at once can corrupt the SD card, and when that happens, cache.xml has always been just under 64kb in size.  It's possible that the ereader can't write more than 64kb to its cache file at a time, and Sony just didn't figure that people would want to dump hundreds of books on in one massive chunk.  My cache.xml file is about 800kb at time of writing so we know the file can get big if it needs to, it just can't get bigger by more than 64kb (theoretically) in a single operation.

== Smaller number of books, but each book is too big ==
I've also corrupted an SD card by combining twelve volumes of manga into one whopping half-a-gigabyte file, because I thought I was being clever.  Oops.  The ereader has to open each file you send it, in order to figure out from its metadata what sort of file it is, and who's the author and what's the title; it probably ran out of memory and crashed while it had its cache.xml file open.  The poor thing's only got 64mb of RAM after all.

== SD card too big ==
Although the PRS-505 can "see" SD cards of up to 16 gigabytes, in practice once you go over 4gb of Stuff on the card, things get really screwy.  This is because the reader doesn't mount SD cards the way you'd expect any standard Linux system to mount them (aye these are linux-powered!), but uses some weird thing that Sony came up with that takes all kinda shortcuts to save battery life.  Since I figured this out (by repeated frustrated experimentation plus trawling long-dead forum threads) I've repartitioned my memory card to 3.9 gigs (leaving half of its space unpartitioned) to make sure that there's no way to go over 4 gigs.

(how would you even go over 4 gigs of books, Dan?  Well, you can fit like a thousand books in one volume of manga and Attack on Titan is really good sooo...)

=== How to avoid it ===
If you're sending dozens and dozens of books, or a few enormous books (comics and manga are especially bad for this), break up the Big Send into several Little Sends.  Send a few books, then eject, unplug, allow the ereader to spin its circle and do its sniffing etc, then once it's done, plug back in and send more.  How many?  I've found thirty or so regular book-sized-books (under a meg each) or four-five manga volumes (like 20 or 30ish megs) in one go doesn't cause any problems.  I've sent ten mangas at a time and had it be fine but, eeeehhhhhh, I don't like to.  Makes me nervous.

=== How to fix it if it does happen ===
You gotta format the SD card and start all over again, sorry.

"Well, I'll just run a disk scan, correct any errors it finds, erase the cache.xml file and let the ereader rebuild it, eh?" yeah no, I had the same idea.  It'll fail the same way, or even earlier because you're asking it to scan even more books.  You could try deleting some books from the SD card and then maybe doing that, but it's unlikely to end up less hassle than formatting the SD card and starting over.  You'll lose your place in your books because your reading position is also stored in cache.xml, but fortunately you've still got all your books in Calibre!

=== How to recover more gracefully than that ===
Take a backup of your whole entire SD card every now and then, including the Sony Reader folder with its cache files!  If your SD card gets corrupted, format it and then restore the backup.  Then you'll only have to start from Some Time Ago, rather than All Over Again - and it's WAY faster to restore a SD card image than it is to send all those books again through that retro USB wire.

=== Update 2025-05-15: it's the SD card ===
Quick update on this issue: since repartitioning my 16 gig SD card down to 3.9 gigs, I've been able to send a thousand books or a hundred volumes of manga at once with no problems at all.  Granted, if I do that then the ereader will spin its circle for like an hour lol.

Interestingly, I was able to send the same 3.8ish gigs of books that previously caused problems - it seems that on big SD cards, issues start to crop up as you //approach// four gigs, whereas an SD card with a single 3.9 gig partition can fairly reliably get filled until Calibre tells you there's no more room (leaving maybe half a meg for the cache file to expand).

The most books I've had on at once was around 1,100.  [[https://www.mobileread.com/forums/showthread.php?t=30849|Here's a guy]] who put over 4,000 on, followed by two forum pages of useful technical information (after six forum pages of people bitching at him about it).

====== What else? ======
Part 4 of this guide is [[oldtech:calibre:manga|a special note on comics and manga]], and part 5 is a [[oldtech:calibre:bonus|bonus bit on how to get stories from web pages and turn them into .epub files]].