ScrapTheChan/CHANGELOG.md

Changelog

0.5.0 - 2021-05-03

Added

• Now program makes use of skip_posts argument. Use CLI option -S <number> or --skip-posts <number> to set how much posts you want to skip.

Changed

• Better, minified messages;
• Fixed inheritance of Scraper's subclasses and its sane rewrite that led to future easy extension with way less repeating.
• Added a general class TinyboardLikeParser that implements post parser for all imageboards based on it or the ones that have identical JSON API. From now on all such generalisation classes will end with *LikeParser;
• Changed file_base_url for 8kun.top.

Removed

• Support for Lolifox, since it's gone.

0.4.1 - 2020-12-08

Fixed

• Now HTTPException from http.client and URLError from urllib.request are handled;
• 2ch.hk's stickers handling.

0.4.0 - 2020-11-18

Added

• For 2ch.hk check for if a file is a sticker was added;
• Encoding for !op.txt file was explicitly set to utf-8;
• Handling of connection errors was added so now program won't crash if file doesn't exist or not accessible for any other reason and if any damaged files was created then they will be removed;
• Added 3 retries if file was damaged during downloading;
• To a scraper was added matching of hashes of two files that happen to share same name and size, but hash reported by an imageboard is not the same as of a file. It results in excessive downloading and hash calculations. Hopefully, that only the case for 2ch.hk.

Changed

• FileInfo class is now a frozen dataclass for memory efficiency.

Fixed

• Found that arguments for match function that matches for image.ext pattern were mixed up in places all over the parsers;
• Also for 2ch.hk checking for if sub and com was changed to subject and comment.

0.3.0 - 2020-09-09

Added

• Parser for lolifox.cc.

Removed

• BasicScraper. Not needed anymore, there is a faster threaded version.

Fixed

• Now User-Agent is correctly applied everywhere.

0.2.2 - 2020-07-20

Added

• Parser for 8kun.top.

Changed

• The way of comparison if that site is supported to just looking for a substring.
• Edited regex that checks if filename is just an "image.ext" so it only checks if after "image." only goes 1 to 4 characters.

Notes

• Consider that issue with size on 2ch.hk. Usually it really tells the size in kB. The problem is that sometimes it just wrong.

0.2.1 - 2020-07-18

Changed

• Now program tells you what thread doesn't exist or about to be scraped. That is useful in batch processing with scripts.

0.2.0 - 2020-07-18

Added

• Threaded version of the scraper, so now it is fast as heck!

Fixed

• Handled situation when OP's post has no comment and/or subject.

0.1.0 - 2020-07-08

Added

• JSON parsers for 4chan.org, lainchan.org and 2ch.hk.
• Basic straightforward scraper that downloads files one by one.

Issues

• 2ch.hk: I can't figure out what exactly it tells as a size and hash of a file. Example: file may have a size of 127798 bytes (125K) but 2ch reports 150 and a hash reported doesn't equal to a computed one.