Git log: commit d653159fe6bdcb60dea7323595d10dc7a8745dfd
Author: Antoine Beaupré <anarcat@debian.org>
Date: Fri Jul 10 21:22:54 2020 -0400
add known failing black environment
That .gitlabci came from undertime, so I think it's okay to keep it
around. I'm just not ready to basically rewrite the entire source code
here just yet however, so silence errors from black for now.
commit e04bbcce82b7025a5ae935fca89cf11a317d6644
Author: Antoine Beaupré <anarcat@debian.org>
Date: Fri Jul 10 21:21:33 2020 -0400
try to support python 3.6, thanks to gitlab CI
commit 0b06b6a9bec205e6df0aad8ebc27643791516856
Author: Antoine Beaupré <anarcat@debian.org>
Date: Fri Jul 10 21:20:45 2020 -0400
skip install on flake8, to make it faster
commit f61034f50f9b21136322378b24b60c53c84a31eb
Author: Antoine Beaupré <anarcat@debian.org>
Date: Fri Jul 10 21:02:24 2020 -0400
run tox on whatever python environment is available
This removes a duplicate source of information about where to store
Python versions. The classifiers remain, however.
This makes the `tox` command (ran locally on my laptop) work without
errors too.
commit 2a675780889a7fada8137b2cd2097a0994c1d443
Author: Antoine Beaupré <anarcat@debian.org>
Date: Fri Jul 10 20:43:42 2020 -0400
recover from feedparser exceptions (Closes: #964597)
In the aforementionned bug report, feed2exec crashes brutally (with a
backtrace, and not completely done) on the following feed:
http://www.agendadulibre.org/events.rss?region=12
The full backtrace is:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/feedparser.py", line 3774, in _gen_georss_coords
t = [nxt(), nxt()][::swap and -1 or 1]
StopIteration
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/bin/feed2exec", line 11, in <module>
load_entry_point('feed2exec==0.15.0', 'console_scripts', 'feed2exec')()
File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/usr/lib/python3/dist-packages/click/decorators.py", line 27, in new_func
return f(get_current_context().obj, *args, **kwargs)
File "/usr/lib/python3/dist-packages/feed2exec/__main__.py", line 124, in fetch
st.fetch(parallel, force=force, catchup=catchup)
File "/usr/lib/python3/dist-packages/feed2exec/feeds.py", line 162, in fetch
self.dispatch(feed, feed.parse(body), None, force)
File "/usr/lib/python3/dist-packages/feed2exec/feeds.py", line 399, in parse
data = feedparser.parse(body)
File "/usr/lib/python3/dist-packages/feedparser.py", line 3965, in parse
saxparser.parse(source)
File "/usr/lib/python3.7/xml/sax/expatreader.py", line 111, in parse
xmlreader.IncrementalParser.parse(self, source)
File "/usr/lib/python3.7/xml/sax/xmlreader.py", line 125, in parse
self.feed(buffer)
File "/usr/lib/python3.7/xml/sax/expatreader.py", line 217, in feed
self._parser.Parse(data, isFinal)
File "../Modules/pyexpat.c", line 471, in EndElement
File "/usr/lib/python3.7/xml/sax/expatreader.py", line 381, in end_element_ns
self._cont_handler.endElementNS(pair, None)
File "/usr/lib/python3/dist-packages/feedparser.py", line 2060, in endElementNS
self.unknown_endtag(localname)
File "/usr/lib/python3/dist-packages/feedparser.py", line 704, in unknown_endtag
method()
File "/usr/lib/python3/dist-packages/feedparser.py", line 1471, in _end_georss_point
geometry = _parse_georss_point(self.pop('geometry'))
File "/usr/lib/python3/dist-packages/feedparser.py", line 3783, in _parse_georss_point
coords = list(_gen_georss_coords(value, swap, dims))
RuntimeError: generator raised StopIteration
I can also reproduce this with plain feedparser with:
python3 -c 'import feedparser; feedparser.parse("http://www.agendadulibre.org/events.rss?region=12")'
So, to feed2exec's defense, this is purely feedparser's fault. Still,
play the defensive programming game and do not let feedparser failing
on a single feed crash the entire run. And even if it would, we should
still tell the user nicer things than a backtrace (although, to be
fair, maybwe we should tell *developers* the full backtrace).
The feed is, at the time of writing, valid according to this:
http://www.feedvalidator.org/check.cgi?url=http%3A%2F%2Fwww.agendadulibre.org%2Fevents.rss%3Fregion%3D12
I am not adding it to the test suite because it is not clear it is
legally allowed. According to this, the agendadulibre.org source code
is free software (AGPL):
https://www.agendadulibre.org/pages/infos#puis-je-utiliser-le-logiciel-de-lagenda-du-libre-pour-mon-agenda
... but this is more worrisome:
https://www.agendadulibre.org/pages/infos#a-nametraitementtraitement-des-donnes-personnellesa
In english, it says that people are allowed to request their personal
data to be taken out, which is a fair policy in terms of hosting an
agenda, but could be painful if I store that data in a git
repository.
So trust me: it's broken, and this fixes it, kind of.
commit d973b91b07cebe4f3b4b60b140bcc43f8f684fac
Merge: 56f345f 16dfa9b
Author: Antoine Beaupré <anarcat+gitlab@anarc.at>
Date: Mon Mar 2 02:05:47 2020 +0000
Merge branch 'master' into 'master'
README: don't let the example config create a folder named '~/Mailbox/'
See merge request anarcat/feed2exec!7
commit 16dfa9bd5cd720193beb1606f5a02ee817548452
Author: Roland Hieber <rohieb@rohieb.name>
Date: Sun Mar 1 15:34:42 2020 +0100
README: don't let the example config create a folder named '~/Mailbox/'
I was confused that feeds didn't show up in my usualy location, but on
further investigation, I found out that I now had a new subfolder in my
home folder, literally named '~:
$ find ./"'~"
./'~/Maildir
./'~/Maildir/'
./'~/Maildir/'/nasa
./'~/Maildir/'/nasa/tmp
[...]
$ rm -ri \'~/Maildir/
rm: descend into directory ''\''~/Maildir/'? y
rm: descend into directory ''\''~/Maildir/'\'''? y^C
...at this point I was too lazy to interpret all the quotes, and used a
graphical file browser to delete the folders :)
Remove the quotes in the example configuration, which does the expected
thing for me.
commit 56f345fd2dc136628e6b37791533ab7d6b892d58
Author: Antoine Beaupré <anarcat@debian.org>
Date: Mon Dec 9 16:55:15 2019 -0500
add empty test cassettes artifacts
Those seem to be created now that we use the betamax session
everywhere, but they are probably harmless.
commit 3b7feeaaa246a977be3a05f7338f8e44e93430f7
Author: Antoine Beaupré <anarcat@debian.org>
Date: Tue Dec 10 13:30:29 2019 -0500
update the comments to match new code again
commit 44e8112497ed673f6069f450211c9f4153718a60
Author: Antoine Beaupré <anarcat@debian.org>
Date: Tue Dec 10 13:23:15 2019 -0500
update copyright date
commit 2ac64d7983b1e4cdf6acea8a7d22d0746823154e
Author: Antoine Beaupré <anarcat@debian.org>
Date: Tue Dec 10 13:22:42 2019 -0500
update documentation to follow latest changes
commit 289f14c0c020fa67267ae6f64531c08355bd5bd6
Author: Antoine Beaupré <anarcat@debian.org>
Date: Tue Dec 10 13:08:24 2019 -0500
update test docs following refactoring
commit 89dabf11fbb1aa7bf59022d82912b37fef80576d
Merge: 3760618 5f90a2d
Author: Antoine Beaupré <anarcat@debian.org>
Date: Tue Dec 10 12:53:27 2019 -0500
Merge branch 'mvc-refact'
commit 5f90a2d46a23a95f085ba92e90810f4bf8f3a39a
Author: Antoine Beaupré <anarcat@debian.org>
Date: Tue Dec 10 12:12:09 2019 -0500
remove useless --config and --database and override instead
By passing the feed_manager, we have complete and coherent control
over those parameters, without having to pass along the arguments.
The downside is we create empty cassettes for all of those, but that
seems like a minor annoyance compared to the code simplification here.
commit 5a3ab3c0f5b110290038fcc7cfce4dac23fb0561
Author: Antoine Beaupré <anarcat@debian.org>
Date: Tue Dec 10 12:06:18 2019 -0500
explicitely override the feed, not the session
This gives us a single override point that is more cohesive and
cleaner.
It will also make other run tests much easier to call, but that's for
a later commit.
commit 9b77ec86bed94bfec6123e86927f13dc01582897
Author: Antoine Beaupré <anarcat@debian.org>
Date: Tue Dec 10 10:27:37 2019 -0500
DRY: refactor common requirements out of tox and setup.py
Requirements are now stored only in setup.py. Before, we were telling
tox to delegate the test runs to setup.py, but this made it impossible
to pass arguments to pytest. By calling pytest directly, we ensure we
can do stuff like `-v` to see which test is failing, and so on.
We do this by storing the dependencies list for tests in the
extras_requires section of the dict, and reuse that in the
requirements file.
It also seems we had three lists of dependencies:
setup_requires: setuptools_scm, pytest-runner, pytest-cov, sphinx
extras_requires["dev"]: pytest, tox, pyflakes
tests_require: pytest, betamax
We converge all of those on the latter, and remove the pytest-specific
dependencies from setup_requires. Presumably this will still work in
package builds, but it's possible the test_requires will have to be
merged into the setup_requires for that to work.
While we're here, we remove the fakeroot -u references. I never use
this and don't know where that's coming from.
commit 3760618c2adb3f7d721dfdebfa3b5b153cc006d2
Author: Antoine Beaupré <anarcat@debian.org>
Date: Tue Dec 10 10:27:37 2019 -0500
DRY: refactor common requirements out of tox and setup.py
Requirements are now stored only in setup.py. Before, we were telling
tox to delegate the test runs to setup.py, but this made it impossible
to pass arguments to pytest. By calling pytest directly, we ensure we
can do stuff like `-v` to see which test is failing, and so on.
We do this by storing the dependencies list for tests in the
extras_requires section of the dict, and reuse that in the
requirements file.
It also seems we had three lists of dependencies:
setup_requires: setuptools_scm, pytest-runner, pytest-cov, sphinx
extras_requires["dev"]: pytest, tox, pyflakes
tests_require: pytest, betamax
We converge all of those on the latter, and remove the pytest-specific
dependencies from setup_requires. Presumably this will still work in
package builds, but it's possible the test_requires will have to be
merged into the setup_requires for that to work.
While we're here, we remove the fakeroot -u references. I never use
this and don't know where that's coming from.
commit 87b91011fe7d17703a455cbed6f5eda8be761186
Author: Antoine Beaupré <anarcat@debian.org>
Date: Tue Dec 10 10:07:08 2019 -0500
reuse feed_manager object in fetch as well
I see no reason why we need to construct a different object in this
specific class, let's just reuse the one already created.
This should make a tiny improvement on the startup latency, but hasn't
been benchmarked. It should otherwise make the code more readable as well.
commit 76db6bef97ed764504c9dd3248d52ddd40d3aedf
Author: Antoine Beaupré <anarcat@debian.org>
Date: Tue Dec 10 10:05:39 2019 -0500
rename feeds to feed_manager in main
The "feeds" appelation is an old remnant of the previous data
structures. Now we do use a FeedManager everywhere and we should name
it as such. It will be easier to grep for it and will more obviously
show what it is.
commit 7ea2deb0f6e22aafe93dee971c102ff6a1bb010b
Author: Antoine Beaupré <anarcat@debian.org>
Date: Mon Dec 9 21:53:25 2019 -0500
make test suite pass again
This was failing because hooking up the cache into the session
completely obliterates our poor old betamax cache. Instead of doing
that, we politely queue the cache layer behind it...
... except that doesn't really work at all, does it. When the betamax
cassette is hit, it does not need the cache, it *is* a cache. So that
change in the controller setup breaks the test_fetch_cache
which *does* expect the cache to "work" (that is, that the body is
"None"), which it isn't when betamax wins.
So we just mark that test as an expected failure for now. The proper
way of doing this would be for the cache controller to be hit first,
and if it doesn't find anything, ask betamax. But I couldn't figure
out how to do this. I asked upstream:
https://github.com/ionrock/cachecontrol/issues/212
It would require some changes in the cache controller code.
Meanwhile, i tried various horrors to work around this problem. I
tried monkey-patching the base class of CacheControlAdapter so that it
would call betamax, that failed in mysterious ways:
https://stackoverflow.com/a/39311813
Specifically, it told me it couldn't call `send()` on a mock
object. Go figure.
I also tried to monkeypatch my way around this by deriving the
CacheControlAdapter class to inject the betamax class as the called
class, but this ended up in some sort of infinite loop and I got lost.
I *could* have let the test_fetch_cache() function skip betamax and
just talk to the cache. But then it would hit the network, and that
would break autobuilders and all sorts of other assumptions.
So fuck it. The caching layer works well enough, and this fixes the
test suite for now.
commit d9298966ae2065df760b37605746740c513f4385
Author: Antoine Beaupré <anarcat@debian.org>
Date: Mon Dec 9 16:54:27 2019 -0500
reuse existing conf and db_path instead of creating new
commit c3b4db038edda84f2331492e3424e2cfd2854d70
Author: Antoine Beaupré <anarcat@debian.org>
Date: Mon Dec 9 16:38:27 2019 -0500
remove class-level sticky session parameter
This cleans up a lot of stuff. Now we can treat the session as a
normal feed_manager parameter. Since there is usually only one
feed_manager in operation at any time, it is basically a static
member. And if it isn't, that's fine too: anyways the underlying
database structures do the right thing here and take care of locking
and everything.
This allows us to merge the betamax fixture into the feed_manager one,
which is necessary to get a (properly setup) session to do HTTP
requests.
This removes the new sessionCache() hack that was required to pass the
right db_path to the session configuration, which was the main goal of
this refactoring.
commit 3ce0f06712866a7a2a33cbacb17a6f21afeed413
Author: Antoine Beaupré <anarcat@debian.org>
Date: Mon Dec 9 16:12:44 2019 -0500
move session and fetching to the feed manager
Having the session and the network code in the "model" makes no sense:
that stuff belongs in the "controller".
Having it there made it particularly difficult to implement the
caching layer, as I had to pass the db_path around quite a bit.
Now we still have this insanity going on, because I just copy-pasted
code around until things (mostly worked), but this will clear the way
for more, and cleaner refactoring.
I will hopefully be able to support other parsers through such
refactoring as well.
commit b951a9dd5dc9a67e3d2bfa1fc0727ef97722cbf9
Author: Antoine Beaupré <anarcat@debian.org>
Date: Mon Dec 9 16:07:55 2019 -0500
add missing cachecontrol dep
commit 1a8faa0eb8218cca2e34fe4764a0e6358e19dbe1
Author: Antoine Beaupré <anarcat@debian.org>
Date: Mon Dec 9 16:07:30 2019 -0500
sync up tox.ini and setup.py version lists
commit c44686a0665b3357905b23dad70063683a8a0f90
Author: Antoine Beaupré <anarcat@debian.org>
Date: Mon Dec 9 15:56:13 2019 -0500
update supported Python release numbers
commit 3a982db376707196f3699e27dbc0cccdea5613f5
Merge: 320f911 783753a
Author: Antoine Beaupré <anarcat@debian.org>
Date: Mon Dec 9 15:55:30 2019 -0500
Merge remote-tracking branch 'origin/cachecontrol'
commit 320f91108585f3b8aa38c46e49052a4c33611552
Author: Antoine Beaupré <anarcat@debian.org>
Date: Mon Dec 9 15:54:45 2019 -0500
run apt update before install
commit 820c828ab4bed9ad210883e91b0c8cfbd6d29edb
Author: Antoine Beaupré <anarcat@debian.org>
Date: Mon Dec 9 15:53:05 2019 -0500
also test on python 3.8, now available in python:latest
commit abdf38b522e01fd5c06def9d1c39d954de3eb0c2
Author: Antoine Beaupré <anarcat@debian.org>
Date: Mon Dec 9 15:51:59 2019 -0500
avoid newer feedparser versions
feedparser 6.0+ removed the FeedParserDict which we depend on:
https://github.com/kurtmckee/feedparser/issues/197
Until we refactor the Feed class, stick with older versions of feedparser.
commit 891c7dee7a7d2e4c8b8632bd119a31ef809791f1
Author: Antoine Beaupré <anarcat@debian.org>
Date: Mon Dec 9 15:51:24 2019 -0500
install python3-dev, required for compiling regex
Not sure why all that junk is necessary, but I want to fix the build.
commit 783753a1c1ee680dd62ab90c76a57f8f1b47b2fb
Author: Antoine Beaupré <anarcat@debian.org>
Date: Fri Dec 6 23:41:24 2019 -0500
forcibly preset the builtin feed session
Without this setting, the wrong session gets initialized in the new
Feed object. Before the caching layer was implemented, this didn't
matter much because those sessions were never called.
But since we did that, the Feed() constructor will usually open and
therefore create the database if the session isn't already set. So set
it to some dummy value that will never work to keep the constructor
from setting it.
commit 0583e4791317800921bbad1eaf589a057ba2d351
Author: Antoine Beaupré <anarcat@debian.org>
Date: Fri Dec 6 23:41:00 2019 -0500
fix broken cache adapter support
We did not need to pass the if-modified-since header. All that was
needd was that we lookup (and return!) the cache value properly. So
also remove that from the database.
The way things were setup, the `get` parameter was returning something
CacheControl was not expecting (a tuple) instead of the proper
thing (a single blob). Our `value` lookup was also failing, and was
irrelevant.
We also don't need to keep the adapter around, it's already present in
the session's adapters member.
commit bf50ba7828d315b4d49e2dfb3c54d958c5f4cddb
Author: Antoine Beaupré <anarcat@debian.org>
Date: Fri Dec 6 16:13:29 2019 -0500
first attempt at using cachecontrol, failing
It seems we need to provide the timestamp, and it doesn't store it in
the database, so it doesn't send if-modified-since headers, so it
fails.
Maybe we are better off implementing this on our own?
commit 837fcd0d21224146a86e10651f567c75451b3682
Author: Antoine Beaupré <anarcat@debian.org>
Date: Fri Dec 6 15:31:17 2019 -0500
small simplification of session init logic
commit edce0e330ba80c078894f81ef691e991822a5008
Author: Antoine Beaupré <anarcat@debian.org>
Date: Fri Dec 6 15:08:54 2019 -0500
enforce commit in context manager unless explicitely disabled
This makes sure we never, ever forget to commit unless
we *explicitely* disable it.
This is also inspired by cachecontrol-sqlite, except the latter uses
False as a default for the autocommit, which seems like a mistake to me.
commit adcec2e4747df06d5b4aa6e61b27d2d352067977
Author: Antoine Beaupré <anarcat@debian.org>
Date: Fri Dec 6 15:06:08 2019 -0500
implement thread-level locking
This is not absolutely necessary as we don't do thread-level
parallelism. But if we every want to switch back to doing that, this
is an elegant way of supporting that.
Inspired by cachecontrol-sqlite.
commit 431e83826e22f172c0969a22cdd4b75ac0148d5e
Author: Antoine Beaupré <anarcat@debian.org>
Date: Fri Dec 6 15:05:00 2019 -0500
split connection cache logic in a separate function, to simplify
commit 3a7de7f580c6c65984b97f06f649504adc99fc80
Author: Antoine Beaupré <anarcat@debian.org>
Date: Fri Dec 6 14:57:08 2019 -0500
remove conn member in sqlite to force use of context manager
commit 5636d42b073ca8ba287f930b061f0f11525d3441
Author: Antoine Beaupré <anarcat@debian.org>
Date: Fri Dec 6 14:54:43 2019 -0500
use context manager for connexions, inspired by cachecontrol_sqlite
commit c837cd1c2931da5245fca2c7e7939735c04379ef
Author: Antoine Beaupré <anarcat@debian.org>
Date: Fri Dec 6 14:45:43 2019 -0500
factor out getter/setters in the base sqlite class
This should pave the way for reusing this class in a caching backend
like cachecontrol.
commit 6749d9c4faace7ce237a7b0a04bdee7cb83eaabd
Author: Antoine Beaupré <anarcat@debian.org>
Date: Fri Dec 6 13:52:16 2019 -0500
rename FeedCacheStorage to FeedItemCacheStorage
This is, effectively, a per-item cache, not a full feed cache. We want
to implement the latter as part of #10 so it makes sense to rename
this first.
This is an API breaking change.
commit 83c36385931458314d29b6743872b6e01d9d84bf
Author: Antoine Beaupré <anarcat@debian.org>
Date: Fri Dec 6 13:47:08 2019 -0500
split large feeds.py into model.py and controller.py
This rearchitecture seems to make sense to me. I would like to keep
those files smaller and that naming will force me to follow that
model/controller distinction more clearly.
Already, the parse/fetch stuff is more "controller" than "model", but
I have to start somewhere.
I also want to implement caching here (#10) so this will make more
sense as more code will be added to model.py, while keeping the
controller separate.
This is an API breaking change.
